How to Prove Your Product is Creating Impact
This is Part 18/18 in the series “How to Build an Innovative New Product or Company,” on the topic of how to measure the value and ROI your product is having for your buyers and users.
Consider a product that helps prevent patient falls in a hospital. It is a bed alarm that alerts the nurse when a patient is shifting their weight, so the nurse can assist the patient who may be trying to stand up, thus reducing the risk a fall will occur.
Further imagine that, before implementing “Fall Safe,” a particular hospital had 5% of patients fall during their hospital stays. That hospital then implemented Fall Safe in 2015, and falls went down in many of the departments. Over time, falls across the hospital went down to 2.5% and held steady.
How do you take credit for holding the fall rate at 2.5%? If your product weren’t in use, would the rate go back up to 5%? What if the hospital claims they’ve done other things since 2015 that contribute to the 2.5% level? What if the hospital is set to renew their subscription with you and wants to know what improvement you’ll make beyond the current 2.5%? In other words, their expectation might be that 2.5% is now the baseline, and your product will need to move beyond that to be considered a good investment.
What if the severity of patients increased during the past year, and their fall rate went up to 7.5%, but you believe it would have gone much higher had it not been for your product?
If you want to affect an outcome metric, you need to communicate that outcome metric to the buyers and users of your product. Ideally, you’d compare it with a target. In this example, how many falls has the hospital avoided this year? What’s the cost avoided from doing so? For each department, how are they doing? Are they meeting their target? Can the product, itself, communicate this, or is it better to have a monthly email or meeting where this is shared and explained?
Create an index that your customers rely on you to provide, that they will refer to often, and for which the use of your product is the primary mechanism they will use to improve it. You may have to create your own metric that your buyers believe is a leading indicator of the actual outcome metric. This is especially useful, if the real outcome metric is rarely measured or conflated with noisy events.
Customer acquisition via marketing is a good example. Companies spend a lot of money on marketing efforts (e.g., visiting trade shows or running magazine ads) but struggle to know whether current efforts are putting them on a path toward the future sales to which they aspire. So marketing firms have created proxy metrics, such as the open rate of email advertisements. It works well, as marketing team members can focus on increasing it every day. They know if they do that well, it will improve total sales, which is the ultimate outcome, but it may not respond on a daily basis to their efforts.
Proxy metrics should:
- Be easy to measure
- Be concrete and stable
- Correlate to/be predictive of the main metric
- Show you how to hone your actions and efforts in the near term
- Adjust as you take those actions
- Be a clear measurement, not a “black box”
- Ideally, be on a scale of 0–100, where 100 is best
Net promoter score (NPS) is a good example; it has been shown to predict lifetime customer value and renewal rates. Can you create a proxy metric that becomes the de facto standard for your industry?
Can you aspire to continuous improvement? Should you?
Continuous improvement, such as the Toyota Production System and airline safety reporting system enjoy, is a noble goal for a product. If you’re ambitious, you don’t just want your users to try your product; you want them to use it, see an improvement in their goals, then use it more, and over time, see it revolutionize a major part of their lives or careers for the better. This can’t happen from a point solution; it requires cultural change.
If you aspire to continuously improve a metric, you then need to consider how your product fits into the larger ecosystem. You may also need a larger client-care team to ensure culture and behaviors are being built and reinforced around the product.
What if your product improves outcomes, but other forces at work muddy the outcome metric, so it’s hard to tell for sure?
One thing we do at my current company is to split our user base into two or three categories at each client, based on usage: superstars vs. on-track vs. lower users. We then look at the changes in the outcomes metrics for each of the three cohorts every few months. If the overall outcome metric is improving for the highest users, is stagnant for the middle users and is deteriorating for the lowest users, you can easily see the impact your product is having, assuming all other forces are affecting the three groups equally.
What if your product improves outcomes, but your users revert that improvement by using the improvement as an excuse to relax other efforts?
Malcolm Gladwell shares the insight that more pedestrians are killed crossing the street at crosswalks than they are at other locations. Overlooking the mathematics of it (e.g., part of the answer is that many more people try to cross at crosswalks), part of the reason is the crosswalks provide a false sense of security. The crosswalks reduce risk, but that risk reduction is then lost due to more careless actions by the pedestrians.
The crosswalk is a “product,” and it is only half of a pedestrian safety solution. It needs public awareness and other supporting systems. For example, statistics on signs posted at each crosswalk warning of the number of pedestrians killed at crosswalks might help reinforce the need for continued vigilance.
Go ahead, and prove it
You have a great product, you have happy users, and you’re making an impact on the world. The final major step is to document that impact and prove it to the world. Focus on the outcome metric you want to improve. Make sure your product and customer success team, collectively, can own as much of the influences on that metric as possible. Work back from that larger outcome metric to create proxy metrics that provide proof of real-time progress. Find ways to show that your highest users are making the most progress. Most of all, keep it simple: You may spend all your life thinking about the details of these outcomes, but the impact has to be clear and unequivocal to any skeptical stakeholder taking a cursory glance.
Also published on strategist.blog, where I share thoughts on building analytics-based products and businesses (“product strategy”). Any books or other resources mentioned or referenced in this article are listed here.
