Migrated from eDJGroupInc.com. Author: michael simon. Published: 2014-11-23 19:00:00Format, images and links may no longer function correctly.
Greed is good! Who’ll stop
our new robot overlords?
Gordon Gekko wins?
It’s All About the Incentives
In my previous article for the eDJ Group I left you, my intrepid readers, with a challenge:
. . . please do me a favor: look around at the technology you use in your work and in your everyday life (computers and toasters alike) and ask yourself, “Could I explain to someone else exactly how this works?”
Had time to think about it?
OK, now see if you can explain how the devices you use in your everyday life work: your computer . . . your car . . . your toaster. Seriously, can you explain how your toaster works? Can you explain it without using wooly mammoths in the explanation? I know I can’t.
My point? Our modern lives are filled with black boxes, things that we understand in terms of the inputs they require (click the mouse, turn the wheel, insert slice of bread) and the outputs we receive (your computer beeps, your car turns, you get toast!). Yet between the input and the output there are a whole bunch of things happening that we can’t see, can’t explain, and – most importantly – don’t actually need to explain to accomplish our desired task. As long as the inputs are understandable and the outputs are what we expect, what lies in between can be completely opaque. I don’t need to know how my toaster works, as long as I get my toast.
So why is the fact that machine learning (a/k/a “predictive coding”) is a black box such a problem? Is it because human review of documents (i.e., an eyes-on-all-docs full review) is somehow more transparent? Of course not. We have study after study of the greater accuracy and effectiveness of review assisted by machine learning (when used properly). Here are two of the most important studies:
The 2011 TREC Legal Track was the sixth since the Track’s inception in 2006 . . . From 2008 through 2011, the results show that the technology-assisted review efforts of several participants achieve recall scores that are about as high as might reasonably be measured using current evaluation methodologies. These efforts require human review of only a fraction of the entire collection, with the consequence that they are far more cost-effective than manual review.
Grossman, Cormack, Hedlin, Oard, “Overview of the TREC 2011 Legal Track”
Overall, the myth that exhaustive manual review is the most effective – and therefore, the most defensible – approach to document review is strongly refuted. Technology-assisted review can (and does) yield more accurate results than exhaustive manual review, with much
lower effort.
Grossman, Cormack “Technology-Assisted Review in E-Discovery Can Be More Effective and More Efficient Than Exhaustive Manual Review” XVII RICH. J.L. & TECH. 11 (2011)
We also have ample evidence of just how subject human judgment is to bias, subtle environmental impacts or even just plain, random extraneous inputs. If you still have any doubts, just see here . . . or here . . . or here (and there are hundreds more just like these).
Counsel that are truly worried that they will miss something on a privilege review can prevent inadvertent waiver by obtaining a FRE 502(d) order. Magistrate Judge Andrew Peck calls a properly crafted FRE 502(d) order “the get out of jail free card” because it requires the return of inadvertently produced privileged information no matter what. Indeed, Judge Peck tells audiences “it is akin to malpractice not to consider having a 502(d) order in every case.”
And thus it comes down to incentives, or as Greg Buckles quoted a law firm partner: “Your ROI for this stuff comes out of my pocket!” Nobody really wants to talk about it in public, but when the billable hour, despite so many predictions over so many years still stubbornly refuses to die, law firms still have every incentive to bill clients everything that they can. Going through Greg’s Analytic Adoption: 2014 Market Report, I found no less than six references (including the quote above) to law firm and senior partner resistance to machine learning based upon fears that it will decrease billable hours.
So, am I here to tell you that this resistance will go away? No, I won’t because that resistance won’t. In fact, it might even get worse. The law firms that are doing well with eDiscovery have no incentive to change. The economy is picking back up for BigLaw, and those who know law firms see this as a reason for the successful firms to further resist change. Even in the face of decreasing litigation revenue and the corporations bringing more work in house (see here – at page 18) it is easier to grab what is available on the market and hold on to it than to be entrepreneurial and risk of killing the golden goose.
So who cares? Not the law firm leaders in eDiscovery. I highly doubt they would have any reason to read this kind of stuff (if you are, please feel free to call me on this – Greg will be happy to post something that proves me wrong).
Yet, corporate counsel seem to care. Yes, I know that Greg’s research found that only 20% of corporate counsel asked their law firms about predictive coding, but I don’t think that anyone will dispute that because they have been demanding strong cost savings from their firms since 2008-2009. In a confused and difficult market for analytics, corporate counsel don’t seem to know exactly what they need. They do know that they need help: legal market analyst firm BTI has reported that 40% of corporate counsel say they want to see firms manage eDiscovery more effectively and aggressively employ the latest technology to reduce costs. Yet, a 2013 BTI study (see slide 25) reported that companies rate law firms effectiveness and handling and managing eDiscovery at an average of 5.9 out of 10 – what BTI calls “a failing grade by any standards.”
This sure looks like an opportunity to disrupt the market. A few years ago I saw a presentation at the Georgetown Advanced eDiscovery Institute by the then-managing partner of an AmLaw 100 firm that was not particularly well known for being an eDiscovery leader at the time. The managing partner’s presentation focused on how his firm had implemented machine learning and provided five clear case studies where the firm had achieved big savings for clients. A few years later, that firm is now thriving, whereas other eDiscovery leaders are now diminished or gone (and blame eDiscovery for it). The firm took a chance and won by giving the clients what they were demanding: effectively and aggressively using technology to reduce costs.
So why not you?
Michael Simon – eDiscovery Expert Consultant – Seventh Samurai
Contact Michael at Michael.Simon@Seventhsamurai.com
eDJ publishes content from independent sources and partners. If you have great information, perspective or analysis to share, please contact us for details.