The Right Number of Questions for NPS Surveys

Practitioners of NPS seem to agree that it is a revolutionary approach for gathering granular, timely feedback from customers. They also think that it’s a useful process for integrating organizational structures with incentives so that front-line employees are eager to listen and respond. But as with any new approach to management, there are plenty of questions and issues that need to be addressed.

One recurrent source of confusion is my insistence on keeping NPS surveys extremely short, just two or three questions. A short survey seems impractical when we all know that determining the score is just the first step. We need to learn not only the absolute score but also what must be done to improve it. So naturally we want to ask more questions.

Let me make a distinction here between two types of diagnostic processes that companies can use to learn what actions they must take to improve their NPS results. The more common approach is what we call top-down, because it is geared to help headquarters staff and execs discover broad patterns and systemic issues that can be addressed at the corporate center. We believe this kind of research—surveys with lots of questions—can provide valuable insights on issues like strategic segmentation or competitive benchmarking. As you would expect, this kind of survey is a standard feature in most Bain client work.

Yet this sort of top-down research typically does little to change the priorities and behaviors of front-line employees in a sustainable way. That is why firms must also develop a bottom-up (aka “operational”) survey process that is geared to the front line. This is the brilliance of the Enterprise Rent-A-Car process; each branch follows up with customers to learn what improvements are required. The same individuals whose behavior must change engage in conversations with customers to understand their score and what changes make sense. It is this bottom-up process that requires short surveys, correspondingly high response rates, and closed-loop accountability. The bottom-up survey is not a research process looking for statistical correlations; it is a conversation with a specific customer performed by an (appropriately trained) employee on the front line.

This operational, bottom-up feedback is a distinctive pillar of the NPS discipline, designed to reinforce front-line employee learning, organizational responsiveness, and accountability. It is indeed revolutionary to create such a feedback system where the front line is responsible for closing the loop with customers—and for taking action. But we badly need just such a revolution to make firms truly customer-centric.

So don’t get caught up in the debate about whether it is OK to have surveys with more than a few questions. The answer depends on whether the survey is part of a top-down research process or a bottom-up operational process. Most companies need both approaches for different purposes. Also, don’t get caught in the trap of merely adding the ultimate question to an existing, and already lengthy, research survey. Unless you have an operational feedback process as well, you’ll have little opportunity for taking action to create more loyal customers.

Companies that do use both surveys, and that clearly distinguish between them, need to be alert to another source of confusion. The Net Promoter score tallied by their bottom-up operating survey will almost certainly differ from the NPS calculated from the top-down research-oriented surveys. The differences are due to sampling variations, lower response rates, timing of surveys, differing levels of anonymity, and different context. Without careful communication and definition, dueling NPS results can undercut the credibility of the operational NPS. My advice: refer only to the results of your bottom-up operational system as net promoter score. Call the top-down research survey results what they are: net promoter estimates.

I would welcome comments and ideas regarding the best language (or labels) to use so that we can minimize confusion. Top-down and bottom-up don’t quite catch all of the salient issues. So please help the NPS community with your suggestions.

Comments

[Glenn]

You make excellent points. I'm going to bookmark this post so that I can refer to it in the future.

Thanks!

Regards,

Glenn

[Alessandra Frota]

We are a B2B company that has implemented exactly this model. Since last July we are doing Operational Surveys monthly and segmented by product line. They are bottom-up and the interviewed are the managers who deal with our delivery team in the day-by-day. Our top-down survey is called “Annual Strategic Survey” and was done in November with the major execs from our clients. Considering all answers, the NPS is the same for both groups (bottom-up and top-down) but if we analyze it according to customer segments we can see significant differences. We are dealing with this result creating different actions plans per segment. The results appoint we are in the right direction.

[Echo]

I also carried out a small scale experiment survey at operational level. First I prepared 10 questions, but when doing it, I found my questions were cut short in the smoothly carried on conversation with end users, and the questions were focused on only 3-4 questions. I was a bit upset about it, but now, after reading your article, came to realize the differences before NPS and NPE.

[Carrie Sue Weston]

Top-down and bottom-up NPS was instituted at Fireman's Fund in mid 2005. The same confusion exists between the two. In 2007 we will reinforce the top-down NPS as the Benchmark as we ask our customers and prospects to rate Fireman's Fund and up to 2 competitors on the same series of value driver attributes. We will refer to the bottom-up NPS as the Operational NPS as we ask our only our current customers their likelihood to recommend us. Given the attention NPS has received at Fireman's Fund, our internal departments are looking to create an internal 'promoter' process ie a proven practice for measuring the satisfaction of our internal service providers. Any ideas?

[JP Layton]

Re: your request for a better label than top down and bottom up, we tend to think in terms of "tactical" for the operational or bottoms up approach...can be performed as an automated email based on contact center transactions; versus "strategic" for the top down approach...typically will be via phone or personal contact.

[Mark Malysa]

On the subject of "bottom up" NPS surveys, do you recommend that the survey is related or triggered by a customer transaction - so the results are more actionable in terms of changing processes? Conversely, would you recommend that the "top down" survey is a sample of all customers whether or not a transaction has occurred recently?

[Chandra]

Hi

My experience is to be careful with sampling at the bottom up level. I have often seen casual conversations with customers not done in using a proper sampling process could lead to wrong priorities. The other problem that I have noticed with frontline staff are their inability to come up with innovative solutions to customer problems. Example: If customers rated a call centre agent low on empathy, the typical response is to identify employees with low empathy. In practice we have noticed that more than 40% of executives cannot make any change immediately after the training and out of the rest 30% are back to normal after a month. A thorough analysis revealed that the profile of these executives was different from the better performing executives and that the problem was not so much in having or acquiring the skill but the fact that they were not suited for the job. This was more of a recruitment issue than anything else. This took us a whole year to figure because the 'empowered' frontline staff along with 'out of touch' managers were beating around with traditional solutions. In a way I think we are expecting too much from frontline staff many of who donot have the capability to carry out a well rounded analysis and come up with practical solutions.

[Shalini Malhotra]

Ours is a B2B company. We have been doing the "bottom-up" or Transaction survey for the last year and a half with an aim to get feedback from the process owner levels. We are now working on the "top-down" or Relationship survey with an aim to get feedback from the decision makers of the company. The difference in the two questionnaires is that our NPS-T is more detailed and has scores attached to each question. In NPs-R we have 3-4 questions - mainly qualitative in nature.