Scott Smith

About Scott

  • Expert at system design and implementation of customer experience management (CEM) systems, Scott Smith, former Director of Product Management for Satmetrix, writes on CEM data gathering, reporting, analytics, and more with the goal of maximizing business integration of CEM systems.

« To Touch or Not to Touch: Avoiding Survey Toxicity | Main | A Sweet Taste That Lingers: Making Satisfaction Surveys Work »

Comments

Sam Mishra

For new product introductions, sampling will work, since statistical methods will yield the exact sample size. Also, if you decide to survey a population who are not necessarily your customers, you will not run the risk of annoying those who are being surveyed.

However, if you want to target your existing customer base, I agree that evening out the load on an annual (or for that matter, semiannual) basis is a good strategy.

In fact, I manage a product line where we solicit feedback through auto-followup emails. We are constantly improving the algorithms so that we don't send too many emails to the users of the system, and we have self-service features which help the users of the system choose the frequency of the emails they would like to receive.

k kannan

Scott,

I like to explore the census approach in my b2b setting. Currently we are conducting a customer sat survey but we cannot see much correlation between the results and the financial numbers, thus most dont show much interest into the results.
My comnpany deals with large and complex deals where the decision makers can be a group of ppl in a tendering committee. Some of them may not even know us especially when we are trying to sell somethign for the first time to this customer. So does it help to get the NPS from individuals who have little or no relationship with us?

Also, if we are only going to ask one or two questions, say the recommend question and an open ended question pn what we can do better to improve, what will be the best mode of survey, face to face, phone or web based. I believe face to face may not be appropriate since the executive customers time is precious and we do not want t o spend long hours of planning to meet him/her just to spend five minutes in askign the questions.

Finally, do you think the recommend question is applicable to the b2b setting? I have not done any research but gut feelings tell that it may not be teh right question for such setting. This is because:
1. What is the motivation in an customer executive recommending a company to his friend, because genrally his friends will most likely be from teh same industry and therefore very likely a competitor. If a company has given a superior services, do you think an executive will recommend this company to a competitor? So it is highly likely that the very satisfied executive will not recommend to his friend.
2. Recomending to a colleague? Looks more likely since he may want the company to provide good services to other business units within the customer organisation, but I wonder how likely. With the complexity of the business, the company must first have th solution for other business units before it can be recommended. Suppose a company provides unique solution to the customer's delight, and suppose the company could only provide solutions to that business unit. If we are asking the head of the business unit if he/she will recommend us to his/her colleague, what would be the possible response?
Therefore, do you think the question "if choosing for the first time" suits better in such environment?
thanks

James Young, Satmetrix

Responses below to your questions/comments, which I place in quotes:

QUESTION: "Cannot see much correlation between the results and the financial numbers, thus most dont show much interest into the results."

ANSWER: Customer Satisfaction is not necessarily a good predictor of revenue or profitability growth. Net Promoter has been proven to be the best predictor of growth. If Net Promoter were adopted as the key business metric (alongside customer satisfaction), then the relationship with business growth would be more apparent. This then provides a robust setting to establishing which elements of the customer experience are the ones which drive value (Drivers of Recommend)

QUESTION: "My comnpany deals with large and complex deals where the decision makers can be a group of ppl in a tendering committee. Some of them may not even know us especially when we are trying to sell somethign for the first time to this customer. So does it help to get the NPS from individuals who have little or no relationship with us?"

ANSWER: The more complex the deals, and the more people the whole sales process touches, the greater the value of understanding your Net Promoter score. This is so because if only one or two individuals are the users and decision-makers, an in-depth discussion is likely to reveal the true reasons for whether those individuals intend to buy, or not buy. However, where there are many individuals involved in a decision-making process, including those who feed their requirements into a tendering process, from all departments of an organisation for example, you need a framework and reliable metrics to understand what the prospective purchaser values, where, who and why.

As to whether the people involved know you or not - they don't have to have a relationship with you or people, but they do need to have a relationship with or associated opinions/views of your company, even if it is at a relatively superficial level. The key point here is that if those influencing decision-making, however small or big, have been at the receiving end of using the suppliers products or services, they will have an opinion on that experience. Capturing customer response insight and anaysing what is of value to them, across all segments, gives the supplier the opportunity to improve the experience and ultimately influence the buying decision.

QUESTION: "Also, if we are only going to ask one or two questions, say the recommend question and an open ended question pn what we can do better to improve, what will be the best mode of survey, face to face, phone or web based. I believe face to face may not be appropriate since the executive customers time is precious and we do not want to spend long hours of planning to meet him/her just to spend five minutes in askign the questions."

Best practice, in my view, is to adopt a variety of channels and methods when attempting to understand and reveal every facet of the customer experience. The fact is that some people prefer some channels to others. However, online web-based surveys are very effective as they produce quick turn-around in-depth results, and if you're wishing to gather rich data in a short time period, this method is probably the best.

QUESTION: "Finally, do you think the recommend question is applicable to the b2b setting? I have not done any research but gut feelings tell that it may not be teh right question for such setting. This is because:
1. What is the motivation in an customer executive recommending a company to his friend, because genrally his friends will most likely be from teh same industry and therefore very likely a competitor. If a company has given a superior services, do you think an executive will recommend this company to a competitor? So it is highly likely that the very satisfied executive will not recommend to his friend."

ANSWER: Research shows that Net Promoter is the best predictor of business growth in B2B environments. This maybe contrary to initial perception. Research shows that a colleague will 'recommend' to colleagues, business partners and personal friends. The power of recommendation is high and if perpetuates frequently across large organisations, the reputation of a company can be communicated very quickly. I'm not certain about whether a 'competitor' friend would recommend, but of course competitors do often share the same suppliers, and people do move from competitor to competitor.

Garry Aldridge

I'd like to explore the sampling vs. census discussion with respect to a real life problem. We have a client with a customer database of approximately 70,000 people.

If we were to introduce a monthly Net Promoter Score Survey using the 'traditional' sampling method, we would probably survey a random sample of approximately 800 customers, which would provide a Margin of Error of +/-3.44% to a 95% Confidence Level.

Assuming an average of 4-5 contacts per completed survey, you would be 'using up' 3200 to 4000 customer contacts per month in order to get your completed survey. By that I mean, customers that you would tag within the database as having been 'contacted' and who should not be contacted for other surveys for at least 12 months.

However, in one of your past postings, you talked about trying to contact 1/12 of the customer base monthly, so that you could achieve a 'census-like' sample over a period of approximately 12 months.

In my example, this would mean more than 5,830 interviews per month, which is fairly costly even with a 1-4 question survey.

Is it your view that standard sampling levels will be adequate or is it your view that companies need to look at much larger samples, approaching an annual census level, in order to make the NPS work?

Scott Smith

In response to Garry's comment above: "Is it your view that standard sampling levels will be adequate or is it your view that companies need to look at much larger samples, approaching an annual census level, in order to make the NPS work?"

Short answer, no. Standard sampling levels are adequate to get a meaningful NPS. Census comes into play when you are looking to operationalize NPS into other business process like account management.

So, The first thing I would ask. What is the goal of the survey?

For a sample strategy, you will get feedback that will help with strategic and process related issues and get a meaningful NP score. With a census approach, you can get the same as a sample, but you also identify unique customer issues. Use the census approach when you want to build a direct relationship with the customer.

For example, in the account management example, you have a customer that delivers a large amount of revenue. The interactions are complex. You need a census approach to manage that. This also provides a framework for an account manager to interact with the customer and provide a 'fact-based' dialog of the issues. This is most appropriate in B2B environments. Also, given the cost, this may be something you only want to do with the relationships are that important. So, you might census your 'top accounts' while sampling your tier two and three accounts.

If the above is the case, I would suggest that you use a census strategy and even decrease the interval - every 6 months contacting the customer - much more expensive especially if these are live interviews. But worth the expense - save a few customers this way and you get a good ROI.

If account management is not one of the major goals - this is more B2C, then a sample strategy would work fine.

The reason for spreading out the sample, once a month is to level the work-load for account managers that need to follow-up and address the issues.

Andrea Carpenter

Hello,

Can you please comment on how best to report margin of error? On page 101 of The Ultimate Question, Fred appears to describe survey response rate as the percentage of people who answer the phone AND agree to answer the NPS question (a large percentage). This is much different than the number of people who answer the phone VS. the number of attempted calls (a smaller percentage). The approach makes a big difference when calculating margin of error. What is the proper approach?

The comments to this entry are closed.