Migrated from eDJGroupInc.com. Author: Greg Buckles. Published: 2010-11-29 11:50:24Format, images and links may no longer function correctly. Consultants and analysts fly a lot. It is part of the job, but not a part that I particularly enjoy in this age of overbooked flights and mandatory security theater. I am lucky enough to have the flexibility to push several December engagements into January so that I do not have to fly while the new security procedures are debated. I like to think of it as waiting for the first service pack instead of upgrading to a new full release from Microsoft. So how does this relate to eDiscovery? Think of the TSA screening process as a search and review challenge. You have millions of unrelated travelers (substitute email here) with a very few bad actors that have to be identified and acted on. This is remarkably similar to what a corporation with an immature eDiscovery process goes through with any major case. The TSA is in the same position as the corporation’s law firm. The corporation dumps millions of email on the firm and then makes the senior partner (politician) responsible for any privileged or trade secret email that accidentally leak through the review.
http://www.schneier.com/blog/archives/2010/01/tsa_logo_contes.html
The TSA and the law firm are both in a no-win situation. They are tasked with creating a zero-defect search/screening process against a large, heterogeneous population. To make it worse, the TSA has demonstrated the exact same over-reaction reflex that we have seen after cases like Qualcomm Inc. v. Broadcom Corp. and Pension Committee of University of Montral Pension Plan v. Banc of America Securities. Where the TSA has implemented mandatory shoe scans and separate liquid screenings, firms have resorted to second and third pass reviews. I have even heard of some firms making post-production reviews to comply with the relatively short window for claw-back requests.
So what can eDiscovery teach the TSA?
- Individual screening/interviews are not scalable or affordable. When you attempt to make an individual determination on every traveler/email, your screeners/reviewers lose the ability to make accurate judgments. Remember when the first high speed review platforms enabled reviewers to bulk mark documents at more than 60 docs/hour? The actual accuracy of reviewing went down sharply with the increased speed unless the documents were grouped by threading, similarity, concepts or other clustering technology.
- Random sampling is rarely effective on any sufficiently large population. Given the relatively tiny ‘relevant’ terrorist percentage, random screenings are more about ‘defensibility’ than reality. The mathematics of this can make your head spin, but purely random sampling quickly becomes ineffective when your population grows out of scale. Those of us concentrating on these search/precision/recall challenges have moved to targeted sampling techniques. For instance, TSA would probably learn a lot more by doing 100% screening for one hour at a single airport than randomly screening 1% of passengers across all airports.
- Technology must be carefully evaluated for effectiveness, user skill required and secondary impact prior to being put into use. Too many times I have seen a firm partner make a snap purchase decision based on one demo of the latest search, filtering, clustering or automated review system. Post case analysis usually reveals that the associates in their cubicles ignored all the little icons, sorting options and other new toys in their rush to maximize their hours. The lack of notice for the new body scanners reminded me of failed records retention initiatives where new Outlook icons and pop-ups were rejected out of hand by overworked employees.
- Smart profiling raises accuracy without discriminatory stereotypes. A good example of this is the way that we used to pull every email to/from any corporate attorney for a manual partner level review without considering their actual business role or the full recipient list. Outbound emails from a corporate attorney to non-firm domains are probably not privileged. My point here is that profiling is real data analysis. Work smarter, not harder.
- Recognize impossible or impractical goals. Review accuracy is fundamentally a risk vs. cost equation. When this decision was left to the firms, they naturally chose the lowest risk, highest cost options. Now that corporate counsel has started to take back this decision, we are seeing more practical, balanced approaches being implemented in collection, processing and review processes. The American tax payers bear the ultimate cost of inefficient screening, but only their representatives can make the risk/cost decisions.
As a former crime scene investigator (CSI), I have the utmost respect and admiration for those who are still fighting the good fight. The TSA officers are the lowest link in the decision chain, just like associates and contract attorneys reviewing documents. They must exercise their best judgment within the systems that they operate. Let us hope that the TSA administrators learn and adapt to the lessons of scale faster than law firms have so far.