Charles Chaffin’s Numb: How the Information Age Dulls Our Senses and How We Can Get them Back on confirmation bias got me thinking about our new generation A.I. review systems. Legal practitioners have long known about the dangers of confirmation bias blinding them to key facts adverse to their client’s positions. I am not an attorney and will not opine on counsel’s duties or ethics. Three decades supporting prosecutions and litigation has given me insight into how the explosion of ESI and common search/TAR practices can trap counsel in a filter bubble of their own creation. Chaffin asserts that the automated confirmation bias mechanisms in social media’s algorithms have fed our increasing political and social polarization. These systems use our attention, likes and click-throughs to build personalized social relevance engines. It made me wonder whether A.I. driven machine learning reinforces the natural biases inherent in our adversarial legal system, retrieving only documents that match the fact patterns sought.
I do not routinely directly manage investigations and reviews these days. However, I prefer to define and document a client’s protocols by riding shotgun with their team through their eDiscovery lifecycle on a few matters. I routinely find lead counsel driving selective collections based solely on their initial theme and theory. While focused initial collections to confirm witness assertions are valuable, they can easily miss critical alternative avenues of inquiry.
A sharp opposing counsel may have expanded the scope with negotiated search criteria in discovery motion practice. They may not have requested every topic at issue if they were confident that they already possessed the key documents. On top of this, the corporate team has become accustomed to wrangling and resisting raw custodial collection requests when they know that only 1-3% will be produced. I see a hidden risk in a 30-40% rich selective collection being routed straight into continuous active machine learning review. The CAL engine feeds the reviewer the items most like prior relevant items as it builds the relevance model. That sounds a lot like how Google, Facebook and others build their engagement and advertising models.
Being aware of potential confirmation bias is the first step in bursting your own filter bubbles. Most counsel will analyze the opposing claims and factual assumptions to build their own responses. Fewer will set up an internal red team to safely test their strategies before high-risk trials. These techniques still focus on the finding the parties knowns. Only the data or new witnesses can surface previously unknowns that can derail matter theories and overturn expensive work product. With both parties focused on validating their positions, who is intent on understanding what really happened?
We have better tools than ever to conduct in-place searches and visualizations that allow the data to speak for itself. Early investments exploring social networks, concept clusters and tracing key document lifecycles can illuminate dark data channels and repositories. These kinds of efforts are neither necessary nor justified in every matter. The key is to have a documented protocol that requires counsel to make the risk-cost decision for every matter. Legal teams and providers should present and translate all the available options such as sampling, keyword expansion and others that avoid the confirmation bias traps.
Greg Buckles wants your feedback, questions or project inquiries at Greg@eDJGroupInc.com. Contact him directly for a free 15 minute ‘Good Karma’ call. He solves problems and creates eDiscovery solutions for enterprise and law firm clients.
Greg’s blog perspectives are personal opinions and should not be interpreted as a professional judgment or advice. Greg is no longer a journalist and all perspectives are based on best public information. Blog content is neither approved nor reviewed by any providers prior to being published. Do you want to share your own perspective? Greg is looking for practical, professional informative perspectives free of marketing fluff, hidden agendas or personal/product bias. Outside blogs will clearly indicate the author, company and any relevant affiliations.
See Greg’s latest pic on Instagram.