Sampling Sizes – No Easy Answers

Migrated from eDJGroupInc.com. Author: Greg Buckles. Published: 2010-03-02 04:00:11Format, images and links may no longer function correctly. As inside and outside counsel struggle with ever larger ESI collections, the question of appropriate sample sizes for quality assurance pops up on the national lists, conference panels and in social gatherings of law geeks. There are many different statistical theories that can be used to calculate the relative probability that the results of a sample set can be extrapolated against the total collection. In simpler terms, sampling is used to define how confident we are in an assertion when we have not or cannot review every single item due to scale, availability, cost or time. This is expressed as the Confidence Interval or Confidence Range. Do not worry, I am not a statistician and have no intention of even trying to translate significance levels, variability parameters or estimation errors. Instead, I will talk about how sampling can apply to the discovery process.

The most common type of sampling used in the discovery lifecycle is called Acceptance Sampling. These methods of quality assurance and quality control were created during WWII when America scaled up bulk manufacturing as a means of monitoring and improving the process. Paralegals and LitSupport personnel are used to performing convenience sampling on scanned documents, processed ESI, and non-responsive items after a review to reassure themselves that the scanner, vendor or review team did not make significant, consistent errors. This kind of sampling is not intended to find every possible error. Perfection is not the legal standard, but a good spot check is a practical good faith effort at quality assurance.

But what if you have the ESI from your first five custodians and counsel needs to go to a Meet-and-Confer with estimates on the percentage of responsive and privileged ESI? Can you just do a fast review on what you have and then extrapolate those values against the hundreds of potential custodians? People do that every day, but hopefully they clearly disclose the parameters of the sample. What you would like to do is to present your estimates with a realistic, defensible confidence interval and error range estimate.

Example: Of the 20 TB potential collection, we estimate that there will be 8% +1.5% relevant ESI with a 95% Confidence Interval.

These are the kind of hard estimates that should be used to argue undue burden or simple scope definitions. So what makes such calculations the province of experts? After all, you can Google “Sample size calculator” and get hundreds of hits like this one for survey sizes. Most public calculators use models with assumptions about the parameters of the potential collection population and the sampling method. In our five custodian collection example, were these key custodians? Did we get all of their potential data sources? Were they all really representative to the remaining potential custodians? How much of their ESI did we review for the sample? All of these questions should be translated into variability factors within the estimation model by someone who understands the chosen model and can back up the reasonableness of that process. This is not a lost cause. Corporations have hundreds of prior matters that can be used for exactly this kind of exercise and early estimations during negotiations should not require a testifying expert, though only counsel can make that risk-cost judgment.

So you can see that how you select your sample items and the variability of the total collection can be as important as the actual number of items that you review. Surprisingly, the confidence level and error mean associated with a given sample size is generally not dependent upon the size of the total population. As seen in the EDRM Search Guide Appendix 2:

So what Confidence Level is required? Again, only an expert and counsel can make the final determination, but a 95% Confidence Level is generally accepted in industrial quality control sampling with reasonably uniform populations. For high risk issues or highly variable populations, a 99% Confidence Level seems to be more appropriate. Before you panic, remember that the standard is reasonable good faith effort and only counsel and the courts can make that final determination. Sampling should be part of your quality assurance protocol, but blindly checking a couple dozen random items is not statistical sampling. So whether you want to measure quality of a process or extrapolate potential metrics, take a bit of time to familiarize yourself with statistical sampling and know your limits.