Migrated from eDJGroupInc.com. Author: Lynn Frances Jae. Published: 2012-02-09 02:00:01Format, images and links may no longer function correctly. Predictive Coding, or Technology Assisted Review (TAR) was the subject of five sessions at LegalTech. Even when it wasn’t the focus of a session, it was often mentioned. Why? Probably because so many of the eDiscovery experts have been busy reminding us of how ineffective our traditional means of identifying relevant documents actually are. As far back as 1985, the Blair & Maron study found that, though reviewers believed that their keywords had identified 75% of the relevant documents, in reality, they had uncovered only 20%. In the more recent 2009 Text REtrieval Conference (TREC) study, keyword searches were even less effective: finding only 9% of the relevant documents. In contrast, the TREC study found that predictive coding techniques, techniques that transmit the decision criteria of expert reviewers across the data set, had an average recall rate of 41%. The most effective predictive coding technique identified 86% of the relevant documents. The advocates of Technology Assisted Review quoted these studies in support of a shift in the review technique norms.
What surprised me was that many of the panelists said that they turned to these types of technologies only when the document volume was too large for human review alone. A 2010 study published in the Journal of the American Society for Information Science and Technology, found that TAR was at least as effective as the most skilled human reviewers. Now, did I miss something, or have we not learned that human review is the most expensive way to find relevant documents? Generally, when we think about the proportionality considerations of FRCP 26(g)(2)(c), “the request, response, or objection is: … not unreasonable or unduly burdensome or expensive, given the needs of the case,” we think of the burden placed on the opposing party. But what of the burden placed on one’s own clients?
So, in two of the sessions, I asked if it is ethical to continue to turn to human reviewers for all except the largest data sets. The answers I received might surprise you. All respondents agreed that it would appear to be more ethical to incorporate these technologies into reviews, except those involving extremely small data sets, say 500 documents per reviewer. They agreed that two obstacles to adoption exist: community acceptance of the existing techniques and lack of legal precedence for TAR. Essentially, nobody wants to be the first to defend these techniques in court.
I’d like to pose this question to the eDiscovery industry: Given the statistics and expense, is it ethical to depend on keyword search and human reviewers for relevance reviews? Or, must attorneys insist on the use of TAR if they are going to certify that they have performed a reasonable inquiry, that that their response to the discovery request is complete, and that it conforms to the proportionality rule?
Lynn Frances