Loading ...
The Honorable Andrew J. Peck, United States Magistrate Judge for the Southern District of New York, graciously allowed me to interview him after the LTNY Man vs. Machine: The Promise/Challenge of Predictive Coding & Other Disruptive Technologies session in which he participated as a panelist. Judge Peck shared the panel with industry luminaries Maura Grossman and Ralph Losey, and moderator Dean Gonsowski. Overall, the session was excellent – very educational, and well organized.
When I reached out to Judge Peck last week to request the interview, my intention was to write a review of the session. I prepared questions and took fast and furious notes during the session. However, between the time the session was over and the time we sat down for a bite to eat and proceed with the interview, I realized that a session review is not really what will benefit the eDiscovery community the most. I decided, instead, to open up a discussion with the community. Directly below are some of my insights and questions on the session, my post-session discussion with Judge Peck, and the hot topic of Predictive Coding/Technology Assisted Review (“PC-TAR”). Do you agree, disagree, have something to add? Did you attend this or other PC-TAR sessions? What did you think? Please post your comments.
Key Word Searches Don’t Work? While I waited for Judge Peck after the session, I had an opportunity to visit with my friend and industry veteran Chuck Kellner. Chuck disagrees with blanket comments that key word searches simply don’t work and the insinuation that service providers are in favor of the key word method for purposes of profit. While a strong advocate of PC-TAR as a major improvement over search through the use of iterative key word development, Chuck was focusing his comments on the intent and recommendations of responsible service providers. He expressed that: (1) experienced, quality, ethical service providers have been motivated by client need to reduce the overall size of review and cost of discovery, and (2) the method of iterative development of key words can be and has been useful and defensible in the past when done properly. Chuck went on to discuss how to develop iterative workflows, sampling, and processes to use key words as a means of locating and managing ESI in the discovery process. We discussed the difference between a solid, iterative process versus “guessing” at key words and simply trudging forward down that path. That kind of “guessing” is what drew the attention of Judge Peck in the Gross decision (William A. Gross Constr. Assocs. v. Am. Mfrs. Mut. Ins. Co., 2009 U.S. Dist. LEXIS 22903 (S.D.N.Y. Mar. 19, 2009)).
This brings me to my next question…
Will PC-TAR Force the Industry Into Better Workflows? The reality of the state of our industry is that there are still a lot of attorneys and litigants that are not subscribing to well-designed (or any) workflows. Judge Peck told a story during the session that highlights this very fact. If you find yourself before Judge Peck, you will be required to complete a Joint Electronic Discovery Submission and Proposed Order, which is Exhibit “B” to the Judge’s Rule 16 IPTC Scheduling Order. In his story, Judge Peck spoke of a case where the parties agreed that they would print all of their ESI and exchange it in paper form. After he denied the proposal, one of the parties filed a motion to reconsider. Unbelievable? Not really – I still see this in practice a lot. More commonly than the paper scenario, I see parties blindly selecting key words without anything to back up the selection of them (such as asking custodians), followed by not sampling them, or processing all data with no filtering at all (such as applicable date range). Neither of these methods demonstrate efficient and effective (or any) workflows.
One of the most common statements we are hearing in discussions surrounding PC-TAR is that if you want to be able to defend its use, you must have a well developed and solidly documented process that includes appropriate levels of sampling and QC. Wouldn’t you agree that this should apply to all ESI review projects, no matter what technology or approach is being used?
Despite evidence (see, e.g., TREC, eDiscovery Institute, JOLT) that proponents of PC-TAR argue demonstrates otherwise, there are many attorneys and litigants that are concerned about the use of such advanced technology and continue to find an “eyes on every document” approach superior. There are also many that believe that PC-TAR may be superior, but would like to see some caselaw on the topic before they are willing to attempt it. (Note: in the session, Judge Peck hinted that he may have a ruling related to PC-TAR in the near future. We will keep a lookout for it and post as soon as we hear more.)
I am hopeful that as a result of these defensibility discussions, those that are not willing to make the leap to PC-TAR at this point might at least begin to develop improved workflow and processes (such as sampling, iterations, filtering , and QC) to their current processes if they are not already doing so. This is the Pollyanna in me. However, the devil’s advocate in me asks the question: if you are choosing not to apply well developed and solidly documented processes in seemingly more simple approaches to collection and review now, why would the PC-TAR discussions motivate you to do it now? After all, the same discussions took place over key word search and there is published caselaw on the topic. What do you think?
Please use the comments section to post your thoughts and questions on this topic and stay tuned for An Interview with The Honorable Andrew J. Peck – Part Two, which will include discussion on the paradigm shift required for PC-TAR, and community education (bench, bar and client).
Any responsible business decision must be made based on sound processes and practices. Frankly, I’m surprised it is taking the legal community so long to catch up. Project management – new? I don’t think so! Documented workflow – foreign concept? Hardly. Clients have been using these practices for decades in their businesses, whether drilling oil wells, manufacturing the latest widget or providing services to their own client base. It is well beyond time legal practitioners recognize efficient and effective work practices and strive to put them in place as they work in the best interests of their clients.
PC-TAR is the latest concept to gain attention. As mentioned in the various sessions at LegalTech this year, this technology is backed by sound scientific studies and principles. It can, and should be, relied on as a business tool – one of many in the lawyer’s toolkit. Ignorance or lack of understanding should not scare lawyers away from learning and applying this technology to their cases. Find an expert, participate in a demonstration, ask questions. Don’t wait until you are forced into adoption and acceptance of these tools by your clients. Test the concepts along side of your linear review and judge the results for yourself.
As a side note, I agree with Mr. Kellner’s comments regarding key word searches. Common sense, iterative workflows, sampling and documented, repeatable processes are the foundation of effective and useful keyword searches. Use them as a part of your complete review and analysis plans.
February 3, 2012 at 12:25 pm
Dawn Radcliffe
Member Type: Corporate | Role: Litsupport | Size: Large (more than 1000) | Years of Experience: 20 | Certifications/Licenses: CLA
Dawn – Thank you for keeping the conversation going. Your comments definitely hit the nail on the head!
Mikki
February 3, 2012 at 3:38 pm
Mikki Tomlinson
Member Type: Other | Role: Consultant | Size: Small (less than 50) | Years of Experience: 21 | Certifications/Licenses: ACEDS
Thank you for your article Mikki. I think it is important to reiterate that successful eDiscovery is a delicate balance and mix of people, workflow and technology. It seems to me that lawyers currently have an unhealthy reliance on people (manual or linear review), thinking that “eyes-on” review is better than a review using technology. There are is also an unhealthy reliance on doing things the old way (manual or linear review) and that there aren’t many consistent or vetted workflows being used to move review systems forward. I think that lawyers want to use technology to help them move forward but they are afraid because there isn’t a case that says its OK to use these technologies (although Rule 502 says it is in the notes) – even though there NEVER was a case that keyword search is an OK way to find relevant data. I also think that lawyers don’t understand the underlying search technologies and are reluctant to use them in a way that puts them and their reputation on the line for a document production. I see movement in this area mostly with corporations choosing to use technology to lower their costs and control their limited resources. I think law firms will move in this area but it will be slowly (until a case says it’s OK) or until their clients push them to do so.
February 5, 2012 at 12:07 pm
ssigler
Member Type: Corporate | Role: Attorney | Size: Small (less than 50) | Years of Experience: 20 | Certifications/Licenses: CA Bar
I find it interesting that there is the perception that keyword searches are somehow bad or inaccurate. I wonder if this has been a deliberate marketing message from the many software providers that have been investing in analytics? Certainly part of the market wants to believe that there is an ‘eDiscovery Easy Button’ that will magically give us quick, easy relevance determination of huge collections. This is the same minority that does not want to have to think about how messy and complicated real discovery is and has always been. So where are all the providers who touted their Boolean search just 2-3 years ago? It is still a solid, transparent tool in our toolbox that can be applied to the benefit or detriment of any process. Good discussion.
February 5, 2012 at 10:42 pm
Greg Buckles
Member Type: Other | Role: Consultant | Size: Solo | Years of Experience: 22 | Certifications/Licenses: court certified expert witness
Greg, since we all worked together on EDRM and had something to do with EDRM getting the Analytics and Metrics sections developed, we can recall how hard all of the industry service providers worked to make sure we had a very strong methodology for development of search criteria through iterative testing, development and sampling. These, in turn, led to very powerful defensibility arguments for many of our clients who when pushed about their culling methods, had winning arguments every time.
I believe that the misperception about search terms comes about through an oversimplified understanding of the Blair and Marron study and TREC and Sedona and from some desire to pull sound bites out of some of the recent whitepapers from Kershaw and Oot and from Judge Peck’s decision in Gross. It is an oversimplifcation of the proposition that brainstormed search terms WITHOUT MORE are insufficient. You must know what you’re doing to get them to be effective and for their use to be defensible.
A thorough reading of all of these sources brings out the true nature of precision, recall, fallout and the important and defensible ways to improve F1, the average of precision and recall. The Blair and Marron study was my first insight into this proposition, which has been in development since, in fact, the mid 1960s. The legal industry began to embrace defensible and iterative development of search criteria when full text search became widely available early in the last decade.
That being said, now the widespread availability of concept-based search engines gives us more power to improve F1. We can recall more potentially responsive documents without the limitations of Boolean logic through concept analytics, and we can suppress the fallout (recall of non-responsive search hits) through the seeding and training and sampling of the results. Putting these technologies together with workflow and methodology (again, it is sampling and iterative testing!), we are getting very effective and very defensible results. Thus, as an industry, we are delivering results that are, in fact, faster and cheaper and better.
I think that the recent industry studies point out two important ideas: 1. Brainstormed search terms ALONE will miss important relevant documents and will recall false search hits that are too numerous and too expensive to review. 2. Human review isn’t all that it’s cracked up to be because of many factors. I do think that many of the technology-assisted review offerings, WHEN APPLIED WITH THE RIGHT WORKFLOW AND TESTING, can yield us and our clients better results. They can help to make the results of the review more consistent, and because of statistical sampling and revision and adjustment, like any QC and QA process that use them, they help us to find errors that humans have made. Chuck Kellner, ckellner@d4discovery.com http://www.d4discovery.com.
February 9, 2012 at 7:40 am
chuckkellner
Member Type: Provider | Role: Consultant | Size: Med (less than 1000) | Years of Experience: 22 | Certifications/Licenses: n/a