Measuring What Matters - Glenn Donovan

Audience expectations seemed quite high as it was standing room only. Taking a practical approach as a result of his practitioner experience, Dr. Vince Nowinski, Principal Methodologist of Satmetrix, began by addressing the who, what, and when of data collection. He explained how the underlying validity and content of the data collected is what will ultimately impact the “actionability” of the information in your organization. (Yes, that is a made up word, btw.)

In addressing the "who should be surveyed," Vince pointed out that it varies greatly in B2B environments from B2C environments. A key element of measuring what matters is “Voice according to value." Vince went on to contrast B2B and B2C approaches. B2C segmentation is used to inform the organization as to the nature of the core customers, those using the strategic elements of what is delivered to customers. In most organizations these are the customers who drive the bottom line. The goal is to optimize loyalty, not to just drive it up.

In B2C settings, it’s important to segment by value too. The wrinkle in B2B is that there are multiple decision making influences who can be fundamentally characterized as end users, influences and decision makers. It’s important to realize that all voices are not equal and to recognize the relative importance of these various roles in the decision making syndicate.

Now on to the issue of sample versus census. Often times, the driver of the sampling strategy is statistical significance. In fact it’s as important that the right targeting and recruiting of respondents is key to making the results believable and usable in the organization. In a B2B setting, census is the preferred approach, whereas collecting a statistically significant sample is the right approach. But what does this mean in practice? In order to deliver meaningful comparisons between customer groups and other segmentation variables, these must be decided first to drive the correct sampling so the resultant analysis can withstand the questions and criticisms that will come about in most organizations.

Another consideration is to put in the right recruitment and communications approach to prevent potential gaming by the generation of a skewed sample. What data to you collect? There is a controversy regarding the number of questions to ask? In order to answer the questions there a couple of key considerations. One is the impact of length of survey on response rates and the second is how to quantify the drivers of Net Promoter scores.

A two question survey will reduce survey fatigue and tend to be more exploratory in nature. Response rates will be higher. Multiple question surveys will deliver more underlying driver data, but response rates will drop. With a two question survey, the analysis of comments is time intensive and category based to perform root cause analysis. Multiple question surveys are less resource intensive to analyze and yields readily to statistical analysis. However, the development of the additional questions needs to sound in order to generate valid results.

How does one select the appropriate method for your firm? It turns out that while survey length does impact response rate. But after digging into the data, the data revealed there was a large difference in response rates with a two question survey from one to thirty eight percent. What are the steps that impact response rate? Engagement is the underlying driver of response rates, much more than survey length. The comparison was a ten to twelve question survey to a two question survey.

The underlying factors that developed included:

• Auditing the contact list
• Communications to clients around the program
• A relatively short survey that is focused and personalized
• Use standard text to avoid spam filters

To continue that comparison, the question of determining drivers and the categorization of comments versus the collection of driver questions yield different results. There were common themes but the ranking of the longer survey yielded more statistically valid results. Comments are frequently what’s top of mind, but it may not reflect actual importance. Promoters tend to make less comments then detractors, skewing the result towards the negative and as a result, driving less information about what delights customers.

Ultimately, your firm's change strategy should inform your approach. Two questions are useful for customer recovery. A multiple question survey will reveal the customer recovery issues and support valid strategic analysis.

Vince is a great presenter. He was peppered with questions for 45 minutes afterwards in a very lively session.