Suhail Khan: The Methodology of Who, What, and When

The first presentation is by Dr. Vince Nowinski, the Principal Methodologist at Satmetrix (co-developer of Net Promoter). He is the guy who works to maintain sound methodological standards with Satmetrix and based on my observations, it appears he is succeeding. His presentation today is entitled “Measuring What Matters – Capturing the Voice of Your Most Important Customers.” Vince’s approach to the discussion is to cover the "who," "what," and "when," as it relates to sampling strategies, response rates and general survey design. This is a hotly debated subject and has brought much discussion thus far at the conference.

One of the underlying messages from Vince was that getting ‘statistical representativeness of your sample may not be the ultimate goal, especially when considering the differences between B2B versus B2C businesses models. The main takeaway I heard was that when in a B2B environment, particularly when focusing on an enterprise account structure, a census approach is preferred when as compared to a more traditional sampling methodology.  Vince was recommending that you identify your key decision makers and with assistance from your field sales organization, obtain survey responses from 100% of these folks if possible. 

Vince then discussed the considerations around survey length, and raised the debate between survey design using “the one number…” versus the more traditional approach.  Data was shown around survey length and response rates, and I was very surprised that the difference in response rates between a very short survey and the longer survey was actually very small. Vince’s point was that response rates were influenced more strongly by ensuring that the contacts were the most relevant in determining your relationship and the possibility of future business.

One last point that I took from Vince’s presentation was how each of these 2 survey design philosophies could the impact the ability to determine key drivers. My key takeaway is that using the shorter survey, with perhaps the Recommend question and 1 open ended question, the ability to identify accurate drivers of loyalty becomes difficult. This is because the short survey provides you very few options for analysis.  The only tool you have available is to dive into the verbatim comments and determine qualitatively the key drivers.  Using a more in-depth survey design will allow you to use statistical techniques to determine drivers, which to me seems like a more sound approach.

I will be interesting to see how this subject evolves over time as we begin seeing some long-standing practices being challenged.

Comments

[Gido]

On an earlier Satmetrics presentation it was shown that the response rates for 2+1 and 10 questionaires are different...60% for the 2+1 and 30% for the 10 questionaire...How does this link in with Vince's statement that there is no difference in response rate between 2Q and longer surveys?

[Vince Nowinski]

I believe the two findings to which you refer are Fred’s (referencing a Bain survey with executives) and my own (which looked across a number of internet-based brands). In our case, the 2 question surveys yielded a pretty wide range of response rates (none, unfortunately, as good as 60%).

I think it’s fair to think of surveys as a type of social exchange—respondents volunteer their time in the expectation of some direct or indirect benefit. A meaningful pre-existing relationship (as is the case with many B2B businesses) can help to boost the response rate, as can a number of techniques to foster a relationship (the most important of which have to do with making the benefits of participating clear to customers).

No doubt the length of the survey has a contribution to response rates, but I’m not sure it is the critical factor at play (assuming a reasonable length, of course)!

Hope this helps!