For a while now, a few anonymous bloggers have been claiming that NPS does not work. One such critic went public with his objections during the Q&A following a recent speech of mine. This was a giant step forward because it opened up the possibility for dialogue and learning.
So what is the argument? My critic claimed that his data proved that NPS did not explain the differences in growth among online retailers. He believed that the core argument for NPS is that NPS ratings correlate with growth —so if NPS doesn’t correlate with growth in a particular industry, then NPS doesn’t work.
There are a couple of misconceptions here. Let’s take them up one at a time.
First, the NPS framework is not founded on statistical correlations, which can never establish causality. The argument for NPS is based on the relationship between survey scores and the subsequent behaviors of individual customers over time. People who rate a company higher on the NPS scale buy more and refer more friends to the company than people who rate it lower.
The first place I observed the behavior patterns of individual promoters, passives, and detractors was at Enterprise Rent-A-Car, which had analyzed customers’ repurchase and referral behavior after they had responded to Enterprise’s brief survey. Bain, Satmetrix, and I then developed a database of individual survey scores and subsequent behaviors across a sample of 4,000 individual customers for companies in a variety of industries, including cable TV, financial services, and e-commerce.
This data reinforced what I had seen at Enterprise. It demonstrated that one question, scored from zero to ten, was sufficient to predict behavior. Moreover, the corresponding behaviors clustered in a pattern that defined detractors (0-6), passives (7-8) and promoters (9-10).
The NPS framework was thus built on a foundation of individual customer behavior. With that structure in place, we then decided to test its relevance to competitive strategy. So we examined the relationships among competitors to see if our customer-level findings were reflected at the company level. We discovered that in the vast majority of industries there is indeed a strong correlation between NPS and growth rates. This is not a simple piece of analysis since it requires not only accurate NPS data but also data on apples-to-apples revenue growth for competitors (corrected for acquisitions, mergers, accounting differences, etc.).
Bain continues to examine industries around the world—several dozen at this point—and has found that the NPS leader on average has been growing at more than 2.5X the rate of competitors. This is very persuasive evidence, but we try to avoid making too much of these correlations. They are merely statistical corroborations of the underlying NPS engine. We relegated these correlations to the appendix of The Ultimate Question for this very reason. They are not the foundation of the NPS theory, merely supporting evidence.
When I asked our critic if his analysis was based on survey scores of individual customers and their subsequent behaviors over time, he acknowledged that he had no such data. Instead, he had looked at correlations of company growth rates for online retailers versus the NPS ratings from a snapshot sample of their customers. I hope he goes back and gathers the data required to test out NPS survey scores from individual customers and their purchases and referral over time.
I also hope that he addresses a second misconception: namely, his assumption that online retailing is the right business definition for practical competitive decision making.
Much of Bain’s work in strategy consulting has depended on defining businesses correctly, so that the economic consequences of practical alternatives can be understood. We’ve found that “retailing,” for example— even online retailing—doesn’t define a business in a manner that is relevant for making strategic choices. Home Depot and Victoria’s Secret aren’t in the same business, even though both are retailers. LandsEnd.com and Brookstone.com aren’t in the same business, even though both are online retailers.
The correlations between NPS and growth stand up only when a set of competitors is defined correctly. Home Depot must be compared to Lowe’s, not to Victoria’s Secret. If Lowe’s had a higher NPS than Home Depot, that would be a meaningful piece of information for Home Depot’s decision makers, whereas a higher NPS at Victoria’s Secret would be meaningless.
The bottom line? In testing the relevance of NPS to your business, avoid starting with correlations. Instead, begin with real behaviors of individual customers over time. Then, when you examine the correlations of NPS and growth rates, focus your analysis on your true competitors.