PREDICTIVE CODING METRICS ARE FOR WEENIES – Part I

Migrated from eDJGroupInc.com. Author: Karl Schieneman. Published: 2012-10-29 09:00:21  I recently read an assessment from a morning networking meeting held in Chicago about Technology Assisted Review (TAR), that those in attendance believed we needed more metrics around predictive coding tools to help end users grasp how to effectively use them.  This assessment struck me as bizarre because metrics are the very essence of TAR.Common metrics encountered with TAR are the “richness” of a collection,( e.g. how many documents are relevant in a collection being reviewed versus the total population of documents in the collection), and “recall rates” or how many documents the TAR system is finding when compared to the bench mark rate of expert human reviewers on the same sample of documents.  In addition to this, the decision of when to stop or to continue training a TAR system, which is likely based on some combination of objective metrics and some human intuition, and the underlying algorithms for taking the training and applying the results to create complex searches to select more potentially responsive documents without additional human review, are all based on math or metrics of some form.Thus, I am not convinced that a lack of metrics is what the TAR fence-sitters are objecting to.  What I hear in my own discussions with the majority of lawyers on the fence about TAR, both in-house and outside counsel, is that we do not know when we have done a good enough job to convince the other side, and even ourselves, that we can stop.  No one wants to make that decision and be scrutinized, without agreed upon metrics or benchmarks to look to.Put another way, the desire for metrics could also be seen as risk aversion because there is a general tendency among lawyers not to be the first lawyer to try a new technique.  In a world where there is not enough tough love unless you were raised by a “Tiger Mom”, I want to emphatically say, waiting for the metrics to appear that will turn predictive coding tools into an easy button is a futile exercise.  It is a great example of what is wrong with lawyers when risk aversion impedes progress.  Looking for black letter law in an area which can never have black letter law is the type of approach which can keep your eDiscovery efforts stuck in the dark ages. This will also put your organization at a greater risk if you encounter an opposing lawyer who can comfortably work with TAR’s metrics to make their own reasonableness arguments.In my next post, I’ll look at this issue in more depth.eDiscoveryJournal Contributor Karl Schieneman