Too Many Metrics
“Counting sounds easy until we actually attempt it, and then we quickly discover that often we cannot recognize what we ought to count. Numbers are no substitute for clear definitions, and not everything that can be counted counts.”
A key reason for a product transformation is to orient teams toward delivering outcomes rather than just shipping features. Product metrics play a key role in this:
Most feature experiments don't achieve positive outcomes. Bigger, more innovative bets — often the source of outcome breakthroughs — are even less certain. Product metrics help you know what worked, what didn’t, and act accordingly.
Knowing the magnitude and distribution of outcomes helps you make better decisions. You can balance risk/reward and start to see patterns across different potential feature areas that improve your expected results.
In the quest to become more outcome focused there's a common pitfall: Teams commit to being "data-driven" and "outcome-oriented," so they start measuring and keep going until they are reporting on… pretty much everything. Usage statistics. Funnel conversion by step. Button clicks. NPS. Thumbs up and shares. Latency. Time spent on each page. Velocity points. Everyone has their favorite metrics, so teams report on all of them — or at least the ones that make them look good.
None of these are inherently bad, but they don’t tell you which is most important.
The product’s key outcome is often missing or not identified explicitly.
The result? An ever-expanding collection of metrics that individually tell you something interesting but collectively leave you confused. When your quarterly or annual review arrives, you can't determine if you've made meaningful progress.
The opposite of no metrics isn't a laundry list of metrics; it's having clear metrics. These help you generate insights and make better decisions.
Metric Madness
I've seen this pattern repeatedly across companies of all sizes:
At early Groupon, when business performance dipped we would form war rooms and each department would chase dozens of metrics. Even with very smart people involved, we rarely gained valuable insights until we developed a clear framework to isolate factors and focus our energies.
One client shared 16 different success metrics for a single initiative! These included separate categories for "metrics for further user ramp," "metrics for A/B test (L1)," "metrics for A/B test (L2)" and "guardrails." They were overwhelmed by their own data and struggled to make decisions because metrics gave conflicting signals.
Another client planned to track 23 distinct metrics for a new workflow. This would have led to fragmented insights—without clarity on the success of the overall initiative.
As with features, it's easy to generate a list of "good" metrics. It's much harder (and yet critical) to choose the right ones.
The Purpose of Product Metrics
Before diving deeper, let's remember why product teams exist in the first place:
“The purpose of a product team is to solve problems in ways our customers love, yet also work for our business.”
Good product metrics support this purpose by giving teams more clarity and translating conjecture into conclusions. They enable teams to make better decisions through a powerful feedback loop:
Inform: Understand how your product is being used to make better decisions about priorities and designs
Assess: Evaluate feature results to know what's working, what's not, and take action accordingly
Refine: Improve the team's general understanding to increase the success rate of future decisions
Building a Focused Metrics Framework
My favorite way to avoid metric overload is defining them systematically using an outcome KPI pyramid, also known as a value driver tree:
Key customer outcome: This sits at the top of your pyramid and represents your product's overall goal. It may not be directly controllable but captures your ultimate measure of success.
Primary drivers: These are the factors that directly influence your key outcome. You can uncover these by asking "what drives the key outcome?" and focusing on primary components.
Product metrics: By continuing to root cause your primary drivers, you reach specific input metrics directly observable in your product.
When done well, you now have a way to connect directly observable product metrics to your ultimate goal: the key customer outcome. This gives you a set of specific data that is very responsive to a feature change, without losing sight of the big picture for why this feature change was prioritized in the first place.
There are additional practices to refine and improve this framework such as:
Seeking metrics with MECE (Mutually Exclusive, Collectively Exhaustive) relationships
Expressing your product’s metrics as equations to show how they roll up and relate to each other
Using ratios instead of raw values to isolate specific changes from external factors such as marketing spend
Avoiding vanity metrics which don't correlate to product value
Choosing metrics that are actionable by your team, since they are intended to inform decisions
They don't need to be perfect, they need to be useful; if you find they aren't working for you: iterate. Just try to avoid "metrics bloat" where you keep adding more and more without maintaining clear relationships between them.
I go into more detail with examples about how to choose good product metrics in this talk from ProductCamp Chicago 2024.
Product vs. Business Metrics
It's worth noting that business metrics like revenue are often not directly captured in product metrics. Business impact is frequently an indirect second-order effect of product improvements.
As Jeff Bezos noted in Amazon's 2009 letter to shareholders:
“A review of our current goals reveals some interesting statistics:
360 of the 452 goals will have a direct impact on customer experience.
The word 'revenue' is used eight times and 'free cash flow' is used only four times.
In the 452 goals, the terms 'net income,' 'gross profit or margin,' and 'operating profit' are not used once."
While Bezos was making a point about customer focus, Amazon understands that business value is derived from customer value: lower prices, improved selection or easier search lead to more orders; better merchandising leads to customers buying related items and thus larger orders; and faster delivery lead to higher repeat usage.
Putting Your Metrics to Work
Once established, clear metrics can deliver immense value across multiple use cases:
Team Metrics
Quantify your team's purpose via their North Star KPI
Provide clarity, focus and motivation during planning and goal setting
Validate if individual feature results are laddering up to improve the key outcome
Product Metrics
Evaluate the direct impact of features
Inform decisions to roll back, iterate, or double down
Test assumptions and improve understanding to refine broader roadmap plans
Business Metrics
Establish correlations between product and business metrics to inform forecasting
Normalize contributions across teams for comparative purposes
Inform team staffing and performance management decisions
All of these relate back to being able to make better decisions related to a prioritized set of outcomes.
“If you don’t measure it, then people can’t see it. If they can’t see it, they can’t appreciate or value it.”
The Bottom Line
Having too many metrics often is only marginally better than having none. Focused metrics unlock immense value by improving your team's decision-making and outcomes.
When you measure everything, it's still difficult to make decisions. Choose clarity over quantity.
Need Help Focusing Your Metrics?
I've worked with dozens of teams to improve their clarity of focus and ability to make data-informed decisions. If you're looking for support in building a focused metrics framework or otherwise make clearer product decisions, I'd be happy to discuss.
