Monthly Archives: January 2024

There is no denying that kCura’s Relativity leads the online review technology market. Too many of our corporate RFP engagements start with, “Besides Relativity, is there anyone else we should consider?” eDiscovery consumers are incredibly risk adverse. They want a sure thing. In the software industry, I call that the IBM syndrome based on the old FUD sales strategy, “No one ever got fired for buying IBM.” Once a brand is perceived at ‘market leading’ it’s purchase becomes defensible. Clearwell rode this wave to dominate the service provider channel market until the Symantec acquisition in 2011. Knowing that Relativity is at the peak of brand dominance puts immense pressure on Andrew Sieja and his team to use the $125M round of funding to make that market perception of security a technological reality. Every startup has a tendency to ‘check the boxes’, i.e. put in bare bones features to meet RFP requirements. They are so busy trying to be everything to everyone that development cycles are robbed of usability and functional depth. Clearwell definitely fell prey to this syndrome during their time on the eDiscovery wave. The kCura team is very aware of the potential pitfalls of being the market leader and they seem determined to avoid them. Here are my take aways from my briefing with kCura last week:

As I conduct my 2015 Analytics research, I am asking interview respondents how they think the trend towards analytics and automation will impact eDiscovery careers and roles in the next five years. I have just finished my first round of interviews and already have a couple interesting tidbits to share. We all know that technology has the long term potential to replace, reduce or transform jobs in any field. My initial respondents certainly felt that automation of technical collection, processing and culling tasks would reduce technician headcount, but not impact ECA, search crafting and other tasks requiring human discretion. Interestingly enough, they did NOT think that PC-TAR review would eliminate full time paralegal or counsel positions. Instead, they felt that the swollen ranks of firm associates and contract attorney reviewers were vulnerable to market pressure instead. Not from PC-TAR adoption, but from proportionate, targeted collections resulting from maturing eDiscovery teams empowered by the new Federal rules and caselaw. You don’t need PC-TAR for a single skilled practitioner to tackle a couple thousand email and files with a modern eDiscovery platform like Relativity, IPRO, CaseData or others. A decade ago, savvy paralegals loaded small custodial self-collections using Summation’s eDocs & eMail module (sidenote: the requirements for that module were done on bev naps at my favorite SF bar – Plouf). Despite all the technical limitations and problems, that desktop software met the self service needs for most small matters without a huge eDiscovery budget. It is my view that the use of analytics on in-place enterprise data to make proportionate, defensible collections will return us to those days for a large percentage of matters. There will always be massive class action, EEOC, regulatory or IP matters that require large scale PC-TAR or linear review, but these should be the exception. Smart providers and practitioners will adapt their offerings and skills to meet the new demands. More insight as my interviews continue. I still have some time slots open for corporate or firm practitioners willing to share their perspective on analytics in eDiscovery, so send me an email or at least take the survey!

Are you the go-to eDiscovery associate or counsel at your firm? Do they drop you into every meeting that contains esoteric terms like email threading, search criteria or clustering? You are wined and dined by hungry vendors and grateful client litsupport managers alike. But how do your partners and peers view your role and future at the firm? Guess what, you may now be pigeon-holed as the firm geek and your partner track may now lead in circles. Why? Because a good chunk of the money you save your clients generally comes out of the partners pockets. Your culling-processing protocol just cut the collection by 80%. That sent your fellow associates scrambling to find other billable tasks instead of coasting through the holidays at 40 docs/hour. I wanted to share these observations from my ongoing analytics adoption interviews. After a recent briefing with David Cowen, I added asked my interview respondents if they thought that analytics would help or hurt their eDiscovery careers. Several savvy respondents highlighted the general resistance to using PC-TAR for actual relevance/privilege decisions. That opened the door to the observation that generally partners are made based on their ability to recruit new clients and expand firm revenue. Very few firms have launched an eDiscovery practice group or actively promote their eDiscovery expertise/capabilities. These firms may reward and respect an eDiscovery all-star partner, but they seem to be the exception to the rule. It may be that the slow transformation of the U.S. legal economy through Alternative Fee Arrangements and new professional conduct rules will rescue associates with a technical fetish from a career dead end.

As I wind down my 2015 PC-TAR research, I have found some interesting trends driven by the new FRCP amendments and caselaw. Just like last year’s survey results, the numbers do not tell the whole story. With 82.5% of respondents (take the survey!) using PC-TAR to review documents, I could easily jump on the marketing bandwagon and say that it is being used by the majority of matters. I would be wrong. Just because a law firm or corporation has experimented with PC-TAR on a couple cases or been forced to use it due to time/volume constraints does not mean that they are using it for most of their matters. Instead of ramping up from last year’s usage estimate (5-7% of matters), many interviews revealed a laser focus on selective collections to produce proportionate, relevance rich reviews by a small inside/outside counsel team.

It started with a question at the last Today’s General Counsel Exchange in Houston that NO ONE answered. What confidence level or stability measurement is ‘enough’ for civil litigation relevance engines? Actually, all of the usual suspects on the speaker’s circuit gave the classic consultant’s answer of, “It depends…” So I asked a similar question in my 2015 PC-TAR Survey that just closed. Surprisingly, most of the respondents gave real numbers instead of all electing the ‘Other’ full text option. 42% said that they generally felt a 95% confidence level was acceptable while 27% thought 90% worked. Survey questions only tell half the story. That is why I have gotten in the habit of backing up my surveys with 10-20 interviews with eDJ Group participating members. I am always impressed and surprised by their acumen and savvy insight. Almost every interview respondent said something like, “Yeah, I would go for 90-95%, but really it depends…” Behind the survey numbers, their reality was much fuzzier and driven by the matter, jurisdiction, opposing counsel temperament and many more factors. Several reported 60-80% confidence when using PC-TAR to assess opposing productions or high speed regulatory requests when trying to appease an agency demand. I have compiled my updated research for a presentation to the Houston Association of Litigation Support Managers (HALSM) on Wednesday and I hope to find the time to create my formal report this week. Some surprising results, but overall confirmation of the trends highlighted in my 2014 Analytics Adoption report.

I trust my readers to take me to task and keep my blogs accurate with immediate feedback. In response to yesterday’s blog on acceptable confidence levels, I received several complementary emails and one that bluntly called the entire survey question and post “wrong-headed”. My expert friend and critic was right on the money when he pointed out that a confidence level or interval is a sampling measurement rather than a measurement of how accurate or stable your PC-TAR relevance model has become through the sampling/training process. I don’t want to dive back into precision, recall, F1 and other statistical jargon. The PC-TAR promoters have spent 2-4 years trying to educate attorneys and lit support practitioners on the wonders of machine learning with limited success. The smart marketers have dumbed down the ‘when to stop’ message and mistakenly promoted simplistic concepts of ‘98% confident’ because they resonate with consumers. I fell prey to this dumb-speak and promise to do better in future surveys and blogs. The question sought to understand whether practitioners had arrived independently at a standard measurement for completeness when using PC-TAR for typical civil litigation responsiveness retrieval. These are the kinds of questions that our consulting clients ask all the time, “How are others setting the defaults on Relativity Assisted Review training?” In fact, sampling confidence level is one of the first decisions that eDiscovery counsel have to answer. Essentially, how big do our sample training sets need to be for an acceptable level of confidence in the resulting recall, precision and stability? That is easy to understand, much easier than F1 stability, overturn rates and the other measurements that actually support the final ‘when to stop’ decision. No one likes the quenticential consultant/counsel answer to every question, “It depends.” But that is what every interview respondent said when pressed. There is no standard measurement of completeness for PC-TAR yet. Having one would speed adoption, but be wrong in too many instances. Thank you all for the feedback, good and bad. It helped further the discussion and clarified an important point. Keep it up!

After a 2-year hiatus I am excited to return to LTNY this year. Over the course of 2015 we had several clients with a new or renewed interest in matter management and legal support operations process improvement. As a result, I will be continuing my focus on matter management vendors at the show this year.

The forensic security software world has always been a bit strange and incestuous when you peek behind the PR curtain. The core code for modern forensic collection software was created by or for law enforcement or other government agencies. The customers were early CSI geeks like me back in the 1980-90’s. We did not have real budgets and were happy sharing hacks, root kits and command line scripts as we struggled to keep up with rapidly evolving data sources. From my perspective, AccessData has struggled with reclaiming their brand identity after acquiring Summation and launching their own eDiscovery services division. They have returned their focus to selling forensic and security software, but consumers do not follow sudden left turns of the brand like ex-analysts. So AccessData has needed something to reinforce the return to core offerings. They have recruited Victor Limongelli to be Chairman of the Board after a decade managing their primary competitor Guidance Software. Wow. More...

Are you planning on braving the icy sidewalks of New York next week? Well Mikki and myself will be there doing briefings, running down client requests and mostly catching up with friends old and new. Once upon a time, you could find our team anchoring the Hilton lobby bar through the days, but remodeling has made it challenging to find a good location to talk shop. Instead of trying to schedule peers and friends, I have decided to adjourn to the Old Castle Pub on Tuesday afternoon (4-6pm) before all the events kick off. So if you are worn out from your first day of sessions and exhibit hall crowds, drop through and visit with us.

If you missed it, I wanted to call your attention to the raw survey results for my 2016 survey on mobile device discovery. If you registered via LinkedIn or you used a work email address so that we can verify your market role (Corporate/Firm/Provider/Government), then you can access all premium eDJ reports, raw survey results, eDJ Notes, etc. by taking a short monthly poll. This converts you to a Participating member in our not for profit research site. I hope to do a full research report updating the state of mobile device discovery soon.