For enterprise on M365 E3/E5 that have been running keyword searches to export data from OneDrive and SharePoint for discovery in 2020, you may have had a problem.
This will be a long piece, so I will try to pack the important parts up front. Last August, while testing the new online Microsoft (“MSFT”) Word transcription feature I stumbled onto what appeared to be an issue with Microsoft365 (“M365”) Core/Advanced eDiscovery (“AED2”) eDiscovery searches targeting OneDrive sources (details below). For an unknown time period prior to November 2020, eDiscovery keyword searches on OneDrive or SharePoint targets in some M365 tenants may have returned incomplete results without any errors or signs of issues. MSFT quietly rolled out a fix prior to year-end that corrected the original issue without an admin/customer alert. Incomplete results in a discovery request search can have serious downstream consequences, so this blog covers what I know and my best recommendations for determining if you were affected.
Key Points:
- What is the issue? Core/AED2 eDiscovery searches using keyword conditions targeting OneDrive or SharePoint sites may not return full results. We did not see the same issues in keyword searches targeting Exchange mailboxes. See last timeline detail for MSFT’s explanation of cause.
- Who was potentially affected? Of six M365 tenants that I, clients and peers tested with keyword searches for known files returned incomplete results, three were affected. MSFT indicated that the fix patch was rolled out globally in November and the issue was resolved in my tenant in early December. Third party applications that use the M365 eDiscovery service (and potentially cmdlets) may have been affected.
- When did this issue start? We know it affected some M365 tenant environments starting at the latest in August, but we do not yet have a known start date. All of my prior retrieval testing of SP/OneDrive for clients has been focused on full site export workflows rather than keyword results validation.
- How can I tell if a search was missing items? As long as the targeted files remain under legal hold, you can run a new search targets/conditions to compare the hit counts of prior results with current results. We recommend that you set the end date to the day after you ran your search if you believe that new responsive files/items may have been added. During the issue investigation we used the SharePoint content search page to determine which searches were missing items. https://[YOURTENANT].sharepoint.com/search/Pages/default.aspx
- If I find a search missing results, what should I do? First, counsel should always be consulted on potential errors/omissions. That aside, define the scope as quickly as possible by deduplicating the new results against your prior search. You can then determine if there potentially relevant items were missed and must be produced. I tend to see very low relevance richness in most upstream M365 keyword searches, which fortunately decreases the odds of any missing relevant results. My final, not-legal-advice recommendation based on my years of experience: err on the side of timely ethical disclosure, even if it is uncomfortable.
Putting Bugs in Perspective
Modern global cloud systems are dynamic environments driven by agile development cycles. This is a change for front line eDiscovery workers who have historically ‘owned’ their collection tools and data sources. The old ‘better practice’ involved thorough acceptance testing on implementation and key quality control tests on every update applied based on the release notes. A dirty secret of enterprise-level eDiscovery technologies is that there have always been bugs that might impact search, review, etc. Service providers manage hundreds of client matter instances and have filled the role of front-line testing and support for the technology providers. Time after time I have encountered data anomalies in cases that traced back to a bug that it turns out had already been identified and patched without having to notify my clients.
The difference in enterprise cloud repositories is that now corporate legal personnel are running their own keyword searches, unknowingly transferring the burden on those internal discovery workers to regularly test and validate the key enterprise technologies owned by an IT department. Unsurprisingly, the need to run QC checks can fall in between the cracks in the silos.
Over the years I have broken far too many provider’s eDiscovery toys. Providers deserve the opportunity to perform a thorough investigation and manage any customer notification. Releasing incomplete information about a potential bug that could affect eDiscovery accuracy or completeness is not something done lightly.
Early in my career I encountered a similar technical issue that resulted in key evidence not being produced to a very irate US attorney. We had good documentation, found the issue and my FBI interview was only mildly painful. “Trust but verify” has been my motto ever since.
The eDiscovery industry has a bad habit of burying our dirty laundry. That habit does a disservice to all peers trying to do the right thing. I believe that MSFT will communicate the full details of this issue and look forward to working with them to make M365 eDiscovery better. We all need the M365 architecture, tools and APIs to meet our legal requirements on the ESI it hosts. My search feeds are full of webinars and whitepapers from IT providers and M365 partners offering their M365 expertise and solutions. Why have none of them found and published this or other issues? Just remember that before you trust them with your eDiscovery.
Detailed Timeline for Masochistic Techies:
All of this started in late August while I tested Microsoft’s new online Word transcription feature. I wanted to see if M365 eDiscovery was smart enough to retrieve the Word document and the associated audio/video file automatically stored in the /Transcribed Files/ folder. The bullets below walk through the rough timeline while skipping a LOT of support sessions and stupid human tricks that I did to convince MSFT support that this was indeed a real, repeatable search error.
As an aside, if you identify future issues, remember that it will save you a lot of time if you document the full parameters of the issue. AED2 now has a debug info button that really helps engineering support get right to your saved searches/review sets.
- Aug 26 – I created three new Word documents testing the transcription function that were automatically stored in my OneDrive along with the three corresponding audio files. The feature embeds the term “Transcript” as part of the formatting, so I used that as a keyword in Core eDiscovery. To my surprise, the search and subsequent confirmation searches returned zero hits after waiting a day for indexing.
- eDJ – M365 indexing performance varies wildly. So do not expect newly created files to be crawled for up to 24 hours. The Core/AED2 interfaces sometimes take extended time to show search, export and other job results. The status detail pages default to zero hits, so always hit refresh or wait for other details to appear before assuming no hits.
- Sep – Frontline support initially thought that my tenant’s indexing service was not ingesting new documents after several hours of tedious required checks. I could hear them clicking through the screens. I ran the PowerShell scripts requested, sent in various reports created and then forced a complete index recrawl of my OneDrive with logging enabled. I was able to find the six target files in the log being successfully crawled. Support tried to tell me it was ‘acting as designed’ and closed the ticket.
- Sep – Reaching out to clients and peers, my test scenario was repeated in 6 tenants. Two of them report similar missing/incomplete search results for known files. Now I knew that it was not just my tenant and it potentially affects my clients. That ups the ante.
- Sep – Luckily, I have a lot of friends in strange places, so I Teamed (yes, it is now a verb) an expert at Microsoft. I was convinced that I somehow had the Core search criteria or target URL wrong. Nope, the search should have worked. He walked me through his AED2 protocol for creating a new case, adding myself as custodian, running the AED2 search and adding the results to a review set. I waited the recommended 24 hours for AED2 to build the custodian ‘deep index’ and executed my new searches. Nada.
- Sep – Opened new support ticket and did the stupid human help desk tricks again. This time they believed me enough to pull in engineering. Engineering thought it might be a UI issue, so had me install Fiddler and replicate all search actions in Core/AED2 to collect a session log. Never heard back, but I assumed that the UI was not the cause of the search issue.
- Oct – Frustrated with the passing time, I tried running searches with date conditions instead. I got a single file hit (not the 6 hits expected). I got the same result in the Compliance Content Search page. Going even further afield, I ran the date and keyword searches in the SharePoint Search page (link in Key Points above). The SP Search page returned all six hits. So now we knew that the index was fine. Support checked file permissions and insisted on rebuilding the OneDrive index again. We are in October by this point. After the rebuild, Core eDiscovery search found a single transcript file keyword hit(out of six). I stumbled across some other search issues in my expanded testing, but let’s keep this focused on resolving the simple keyword search.
- By mid-October I am seriously concerned/confused. This has come to product management’s attention. I have to say that the Microsoft team was responsive and immediately escalated the support level. No more stupid human help desk tricks. The new consultant had me run PS scripts to pull all the diagnostics and parameters. Engineering went off to do what engineers do with occasional requests to retest my searches. We crept into holiday season and everyone went quiet for a while.
- Nov – Checking back in with the PM team, they indicated that a fix had been rolled out in November and it should propagate to my tenant at some point. I requested an explanation of the cause, scope, etc. It looked like Microsoft was working to pull together a customer notice. Having had to do this fire drill when I was a Symantec PM, I understand the effort and approvals required in these situations. Thus I let it ride through the new year and got caught up with client deliverables.
- Late Jan – PM team provided the general context and potential cause of the issue. As I have observed, MSFT has been making rapid improvements in the architecture of the search index and storage. In this scenario their investigations indicate that the core eD/AED2 service locations of a few tenants fell out of sync during recent storage migrations. MSFT believes that the impact was very limited, that they addressed the affected tenants and have made investments in active monitoring that should ‘make it more foolproof going forward’.
Wrap Up
I was fortunate to have had support from the MSFT product management and engineering teams after going through all the diagnostic rounds with front line support. From everything I can tell, the fix was developed and rolled out globally in a timely manner once the issue was defined and escalated. These kinds of search/retrieval bugs show up in new releases all the time. Most are quickly caught, fixed and customers notified so that they can remediate any potentially affected matters/searches. I would love to hear whether any of your late 2020 searches were affected or if you have stumbled over your own bugs in any of your eDiscovery technology. The providers need our feedback to improve their products and we need more transparency to minimize the growing pains.
Greg Buckles wants your feedback, questions or project inquiries at Greg@eDJGroupInc.com. Contact him directly for a free 15 minute ‘Good Karma’ call. He solves problems and creates eDiscovery solutions for enterprise and law firm clients.
Greg’s blog perspectives are personal opinions and should not be interpreted as a professional judgment or advice. Greg is no longer a journalist and all perspectives are based on best public information. Blog content is neither approved nor reviewed by any providers prior to being posted. Do you want to share your own perspective? Greg is looking for practical, professional informative perspectives free of marketing fluff, hidden agendas or personal/product bias. Outside blogs will clearly indicate the author, company and any relevant affiliations.
See Greg’s latest pic on Instagram.