Trash or Treasure? Expiry vs. Big Data

Migrated from eDJGroupInc.com. Author: Greg Buckles. Published: 2011-10-13 17:24:30  Big Data was a top theme in StoredIQ’s Industry Advisory Board last week. The StoreIQ team believe in the ongoing transformation of enterprise digital landfills from trash to the treasure trove promised by business analytic mining of “Big Data”. The fundamental idea is that immature repositories of unstructured ESI are a liability, while a mature content lifecycle environment (people, process and technology) makes that wealth of information work for the benefit of the company. Let’s face it, storage is relatively cheap compared to 10 years ago. We spent a fair amount of time discussing the key factors to calculating the ‘true cost of ownership’ for a mythical gigabyte of Office files, but the cloud market has already given us a solid baseline cost. Amazon S3 and their competitors can make a profit at 6-14¢/GB/month, so that pretty much confirms that actual storage is cheap.

So why not just keep everything? Storage may be cheap, but the technology and effort required to organize, categorize, search, retrieve and analyze Big Data is not. StoredIQ, Autonomy, Recommind and all the other enterprise search/analytic platforms are betting that Legal’s reflexive call to “Delete it all!” will ultimately fail or that they will become the pruning shears of retention management. The cost to review for relevance and privilege is the true driver of the recent push to implement legal holds and expire non-records from the digital landfill. If analytics such as predictive coding, auto categorization, clustering and more can break the stranglehold of manual review then I believe that the pressure to delete will abate.After all, most users have always been hoarders of their communications and files. They know that the keys to doing their jobs are hidden in all that digital trash. They will be fine with the system cleaning up news, lunch invites and other casual communications, especially since they are already moving from ‘formal’ email to FaceBook and other social media. Corporate IT and users need the documents and meta-knowledge (extracted business intelligence) to do their jobs. The 2011 IBM Global CIO Study showed consistent imperatives for CIO’s to leverage expanding digital assets.  “An overwhelming 98 percent of those CIOs said they would lead or support efforts to simplify internal key processes. A full 95 percent said they would lead or support efforts to drive better real-time decisions and take advantage of analytics.” (page 35 of the report)So IT and the users want to keep their data, but Legal wants it to go away before it becomes subject to discovery. We can all see who is going to lose this fight in the long run. Legal can and will push to clean ‘non-records’ out of legacy digital landfills in the short term based on the current high cost of litigation discovery. Meanwhile, CIOs and business leaders are investing in search and analytics that will eventually automatically categorize the vast majority of ESI, effectively transforming ESI into business records.Tom Bishop, the new CTO for StoreIQ, had a great quote/comment about the transformation of the currency economy into the information economy that tied into their vision of how Big Data would become the foundation for ‘Big Business’ in the coming years. It really focused my attention on the inherent conflict between expiry and the value in digital assets. So many of our recent corporate consulting engagements on legal holds and managed destruction have been driven by the costs of discovery that I feel that corporate Legal has lost sight of the treasures buried in their digital landfills. Now that the economy is slowly digging itself out of the recession (from what I am seeing), I hope that the new wave of scalable enterprise analytics will present such a competitive advantage that it will force corporations to reevaluate this reactionary impulse to delete where possible.