Market Research Needs Less Statistical Analysis
February 8, 2010 15 Comments
I’ve noticed a very important shift in market research. Leading-edge firms are relying less on traditional statistical analysis and more on what I’m calling “Contextual insight” which I’ve defined as
Observations drawn from data that resonates with an understanding of the business
To understand the difference between these paradigms, let’s look at an example: A company with a quality problem. Here are two scenarios:
- Scenario #1: Statistical analysis. The market research organization analyzes warrantee data but it takes several months to collect the information. Unfortunately, the problems aren’t coded very effectively so it takes even longer for there to be enough feedback to isolate a specific problem. Six months after the problem started occurring, the market research organization identified an issue that they felt was statistically significant. The product manager contacted the supplier and resolved the problem.
- Scenario #2: Contextual insight. A product manager had a nagging worry about the vendor who was supplying a piece of plastic used in one of his products. It passed all of the quality tests, but he was still concerned. He gets daily feeds of any quality issues with his products. After a few returns that he could interpret as being a problem with that plastic piece, he immediately contacted the supplier and resolved the problem.
Here’s the basic idea…
Businesses are full of scenarios where contextual insight is extremely important; call center managers who have concerns about agents to a Web designers who worry about parts of the site.
While the traditional market research efforts are not going away, the rise of contextual insight will push market research organizations to change how they operate. Instead of delivering periodic reports, new-age market research organizations will need to:
- Develop more ongoing collection mechanisms
- Process data in a more continuous fashion
- Provide tools for setting up personalized alerts
- Tailor reports and analysis to for different roles across the organization
- Trade-off deep analytical skills for business skills
The reality is that most people in an organization have very limited visibility into what’s happening in the marketplace. Providing them with relevant data (that isn’t necessarily statistically significant) will help them make better, faster decisions.
The bottom line: Organizations need to enable more contextual insight.
About Bruce Temkin, CCXP
Customer Experience Matters® 
This is absolutely true. I just think that there are very few people that have the ability to look at data with an analytical mind, and give contextual insight. I think often you find analytical people going the analysis route, and big picture people giving insight that has no foundation in the data, is lacking objectivity and is often “what the client wants to hear”.
All I’m saying is that I think there is a skill and talent that needs to be grown and developed to be able to do what you mentioned above. Those that do get it right will be able to give immeasurable value to their customers.
Thanks
I agree – contextual insight often points you in the right direction. Once you are, you can perform root cause analysis and get to a solution quickly.
Great post! The distinction between contextual insight and statistical insight is a very important one.
The increasing democratization of data over the past many years has been a very beneficial movement, but has seemingly led to an unforeseen side effect. Primarily, the over reliance on advanced analytics and statistical analyses/models to help make meaning of this data, but in ways that are often developed and applied in the absence of the Contextual Insight that Bruce describes (and in many cases, in a with an abstract focus that is often difficult to understand by the very decision makers it should be informing).
I wouldn’t characterize the distinction between Contextual Insight and Statistical Insight as one of the ‘either/or’ variety, but would generally view Contextual Insight as a necessary component that must inform statistical analysis and insight. When used appropriately, data should be empowering. And, contextual insight is often more than sufficient to empower decision makers to take action.
Hi everyone: Thanks for adding to the discussion about Contextual Insight. I agree with Kubos that there will be some new skills needed and with David that statistical analysis doesn’t go away. Keep up the dialogue!
100% agree with this post, Bruce.
I think our job as researchers is to understand our clients’ businesses well enough that we can pick up signal from noise much more quickly, but retain the integrity of making recommendations that are grounded in evidence.
If we’re thinking of survey research, the other angle is that we need to design questionnaires that will give clients actionable insight rather than good information (i.e. “lets do more of this” or “let’s never do that”, not “hmm, interesting”).
This one hits close to home, previously at another company I was conducting a lot of contextual research to understand our breakdowns in customer service… doing this was putting me in opposition to the statisticians who thought that surveys and quantitative methods were the answer…
End of the story was… statisticians were able to say there was a problem, I was able to speak to the severity, and provide design solutions to fix the problem…. Yet due to politics my insights feel on deaf ears… wish this would have been authored a year ago…
Bruce
The first scenario is not true marketing research – it is a “debugging” or post-mortem analysis to find the root cause of the problem. The flaw you identify with this are bad data collection procedures, bad data and lack of leading quality indicators. It should also be pointed out that the company lacks relevant upfront quality control procedures that allowed flaws to get through. They need to ask what right questions we are not asking and how relevant is our testing procedure for supplier parts and fix those.
The second case almost borders on gut feels. If one Product Manager has an opinion another might have completely different view. The views are based on their background, training and their belief system. If these are not based on hard data to back up then the loudest one prevails.
I disagree that we need less statistical analysis but better, faster and relevant analysis with the application of minds to make profitable decisions. I want to quote here what Michael Lewis (author of Money Ball) wrote in a NYTimes magazine article on analytics in basket ball:
—
The numbers either refute my thinking or support my thinking,” he says, “and when there’s any question, I trust the numbers. The numbers don’t lie.” Even when the numbers agree with his intuitions, they have an effect.
“It’s a subtle difference, but it has big implications. If you have an intuition of something but no hard evidence to back it up, you might kind of sort of go about putting that intuition into practice, because there’s still some uncertainty if it’s right or wrong.”
—
Thanks for continuing the discussion on Contextual Insight. It’s resonating with most people except Rags, who has been taking a contrarian view on several of my posts (which I actually appreciate, because it helps to further the discussion).
First of all, I’m a big fan of Moneyball (I’m actually scheduled to speak right after Billy Beene at an event in May). But when it comes to companies, I always try to deal with the realities of how I see them operating. Too many decision makers (which are most people in a company) have woefully too little information. So getting some customer feedback into their hands, even if it is less than statistically pure, can help the company stay much more aligned with the marketplace.
That doesn’t mean that game-changing analysis like you find in Moneyball should go away. It’s not an “either-or” situation.
Pingback: FOYER für engagierte Marktforschung » Kein „Entweder Oder“ – eher ein „Mehr davon“
Great post. I also feel that we need less statistical analysis. Numbers can be very empty at times.
Pierre Gauthier
http://www.qualitative-research-canada.com/
Pingback: Co-Creation Is Not A New Rocket Science « Customer Experience Matters
Pingback: 10 Reasons Market Research is Critical to Social Media | a thousand cuts :: adam cohen's blog
Pingback: SMEDIAM – The Social Media Channel for Growing Your Business Online » 10 Reasons Market Research is Critical to Social Media
Pingback: Customer Insight and Action (CIA) Platforms Emerge « Customer Experience Matters
Insight is the point of illumination you get when you consider the customer from different points of view. This point of view can come from several places but increasingly I feel it is coming from the combination of data sources, research, contextual, statistics, observations etc. It is this collaboration of the people with the different sources of data that drives the insight.
Just using one piece of data is not enough, just one department is not enough, collaborate and immerse yourself in the customer experience – that is how you get insight.