LTNY 2010 Part 3 – Secret Shopping, A Failed Experiment

Migrated from eDJGroupInc.com. Author: Greg Buckles. Published: 2010-02-11 13:48:07  This year I tried an experiment in my continual quest to gather as much information from the Discovery Maze (LegalTech New York Exhibit Halls) as possible. The experiment was a bust, but there were lessons in why and how it failed. The idea was to recruit corporate and firm managers and specialists with 5+ years experience to be ‘Secret Shoppers’ on the Exhibit Hall. This way I could try to reach more of the floor and everyone in the network would share their feedback on a set of secured web pages.Like many good ideas, it was better in concept than reality. After contacting over one hundred clients and contacts with the requisite experience and perspective, I determined that only 20-25% were actually attending the show. Wow. Everyone attending like the idea and wanted to see the feedback, but not all were sure how much time they were going to spend with the Exhibitors. I said, “Just do what you can.”As the event approached, I noticed that most of the demos scheduled by clients actively shopping for software were located in adjacent hotels like the Sheraton or The London. That made it difficult to schedule back-to-back demos without running continuously late. I also noticed several providers who bailed on running a booth altogether and just ran a demo suite. This might help explain why we lost the 3^rd floor of the Exhibit Hall this year. Alternatively, my recent web crawl showed that we lost up to 10% of providers in 2009 (see my recent post) and hearsay has it that ALM did not drop booth prices for this year. That could easily explain the flight from the Hilton.After my first demo, I realized that the rest areas beside the bathrooms had been eliminated this year. That meant that there was no easy place to sit down and write up my impressions, much less get online to fill out the web form that I had labored so hard to create. That left me taking notes on a Moleskine™ pad while standing at the booth. This is actually makes booth personnel very nervous, especially when they realize that you are some kind of journalist blogger.Pretty soon, simple questions like “What index do you use?” elicited a scramble to find someone authorized to speak to the press. I actually had folks step away from the booth to huddle before giving answers, especially when the answer was, “We do not give pricing at the show.” More about pricing in the next post.I do not want to give the impression that most of my experiences on the floor devolved into obfuscation and frustration. I was there to get hard facts on new features and see the workflow/interface. The booth personnel had their value proposition spiel down pat this year, but they frequently faltered when I cut to the chase and wanted to know and see how it really worked. They would grab the closest director, VP or other C-level to answer the questions. It was apparent that providers expect a relatively unsophisticated audience for the walk through attendees. They anticipate that most of the more experienced buyers will set up a personal demo. That being said, there was usually at least one Subject Matter Expert (SME) around who could jump on the grenade.So this year’s Secret Shopper experiment flopped. Despite good intentions, my recruits said that it was just too hard to make ‘real sense’ of the products from the show without a private session. In the words of a particularly sharp client, “just wandering the booths was nearly worthless.” I know that I had to work hard to get real information on the show floor. I still like the idea of sharing product impressions and feedback from experienced professionals in the trenches, but I am afraid that Legal Tech is not the right venue to gather than information.