Migrated from eDJGroupInc.com. Author: Greg Buckles. Published: 2017-08-13 20:00:00Format, images and links may no longer function correctly.
Whether we are talking about custodial preservation, journal archiving or any collection/export technology, my moto has always been, ‘trust but verify’. We all learn from our earliest mistakes. Mine was trusting some Microsoft consultant’s that helped me build an email search tool to respond to the first Enron related subpoenas. That drove home the lesson that every environment, data stream and software version need validation testing and ongoing QC checks. Believe me, I hate finding bugs or non-compliance at my clients. I know that I will end up having to write the plain language explanation of any potential data loss or incomplete productions. Enterprise IT departments generally have a lab environment and perform documented acceptance/compatibility tests prior to moving a new application/version to the production environment. Too many legal and compliance professionals have relied on their providers without performing similar validation tests with known data sets and consistently checking different phases of processing, review rules, redaction settings and productions. Now that enterprises are migrating live data under legal hold to Office 365 or other cloud platforms, testing is critical to demonstrate your reasonable faith in these ‘inaccessible’ systems. By that misappropriated term of art, I mean that you no longer have direct access to the storage and databases. So what are some of my basic validation test?
- DADO – Data In = Data Out
- Ingest known data sets, run reports, reconcile exceptions, export everything for comparison
- I recommend running both an external test collection like the EDRM dataset and a known set of internal data (generally from a prior matter).
- No system will process EVERY possible data type, so your job is to identify the exceptions and communicate them to counsel/users.
- Data Integrity – Do No Harm
- Get metadata and hash values of collection and compare against values after processing and export. Dates/times and ownership are the usual issues, but I have found systems that inadvertently altered contents of dynamic or macro based files when doing the text extraction.
- Double check the Last Accessed and Modified Dates on your source collection.
- Consistency Checks – Same Question = Same Answer EVERY Time
- Run and rerun collections, processing, searches, tallies, reports and exports from different user accounts, browsers and sessions on the exact same collections.
- For search, change the order of any criteria to verify subtle index tokenization issues. Amazing how a search can return different results when you change A+B to B+A.
- Cross System Comparisons – More Checks = Better Checks
- Use competitive, known products to run comparison searches, processing and collections on your known data sets.
- For very large, diverse collections the results may not exactly match up, but they should give you consistent result averages to help spot gaps in your own system. Remember that there are many systems like DT Search or others that have a 30 day eval license.
- Key Function Tests – Prioritize Tests Where Data Can Be Lost/Missed
- Walk through your workflow and note anywhere that data could be changed, deleted or left behind.
- For each key function identified, test with different security roles and parameters.
- A good example is testing the application of legal holds prior to enabling expiry within your email environment. If a mailbox is under hold, have YOU tried to delete items yet? What about if you are moving from Microsoft’s EAC driven mailbox litigation holds to the new in-place holds applied through the new Compliance Center? They are very different hold mechanisms, so deserve their own validation tests.
Seeing ‘how the sausage is made’ as an enterprise software product manager made me completely distrust tech company QC/QA protocols. Do they test before release? Usually. Some giant companies seem to use early adopters as defacto beta testers for new versions/products. Legal technology professionals are held to a justifiably high standard. So test your tools and be able to defend your reasonable reliance on them whether on-premise or in the cloud.
Stay skeptical my friends!
Greg Buckles wants your feedback, questions or project inquiries at Greg@eDJGroupInc.com. Contact him directly for a free 15 minute ‘Good Karma’ call. He solves problems and creates eDiscovery solutions for enterprise and law firm clients. His active research topics include analytics, mobile device discovery, the discovery impact of the cloud, Microsoft’s Office 365/2013 eDiscovery Center and multi-matter discovery. Recent consulting engagements include managing preservation during enterprise migrations, legacy tape eliminations, retention enablement and many more.
Greg’s blog perspectives are personal opinions and should not be interpreted as a professional judgment. Greg is no longer a journalists and all perspectives are based on best public information. Blog content is neither approved nor reviewed by any providers prior to being posted. Do you want to share your own perspective? eDJ Group is looking for practical, professional informative perspectives free of marketing fluff, hidden agendas or personal/product bias. Outside blogs will clearly indicate the author, company and any relevant affiliations.