Overconfidence in Product Results
“No problem in judgment and decision making is more prevalent and more potentially catastrophic than overconfidence.”
Many teams falsely treat product development as a fairly deterministic process:
Periodically capture feature concepts from executives, customers and PMs
Give these ideas an impact score e.g. RICE to decide what to build first
Provide a date after a short discussion with engineering
Launch the feature on or before the date promised, cutting scope if needed
Move onto the next row of the RICE spreadsheet
Each of these steps has flaws, which are well covered in Marty Cagan’s article Product Fail.
In my experience, many of the biggest issues are directly or indirectly related to a dangerous and misleading assumption: that these features will achieve their expected outcomes. In reality, somewhere between 1 in 3 and 1 in 10 features will be measurably effective.
Once you recognize this, you will want to rethink your approach to software development.
Here I’m going to focus specifically on 1) overconfidence in feature results; 2) bad decisions product teams make as a result of that; and 3) how you change once you accept reality.
General Overconfidence
Even outside of innovation, people tend to be overconfident in their judgements. This is true whether they are predicting future events such as stock prices, factual trivia, travel times or how many tasks they can get done today. For some people, being consistently wrong doesn’t change their level of confidence, perhaps because they are not even aware of their gaps. In some psychological studies, reading more information increases confidence – but not accuracy. Hope springs eternal.
I’ve seen this phenomenon described as being “sometimes wrong, but never in doubt.”
Sound familiar? Don't worry, we all do this.
There is a quick – and fun – test you can take online which illustrates this and gives you some data to help calibrate your confidence. Confidence Calibration
Product Development Overconfidence
Product development feature success rates are significantly lower than most people expect. In many cases, companies don’t even habitually evaluate feature results, based on the false premise that delivery means mission accomplished.
Data shows that most experiments don’t achieve a meaningful improvement:
Google & Bing: 10% - 20% of experiments generate positive results.
Microsoft: 1/3 are effective; 1/3 have neutral results; 1/3 have negative results.
Groupon (where I used to work): 1/4 of experiments generated a statistically significant lift.
Optimizely: average win rate of 12%.
Despite this, most companies are not consistently evaluating results post-launch. I did an informal survey, asking how often people measure feature results. The majority of respondents said they evaluate features less than 25% of the time.
Overconfidence without evaluation is a dangerous combination.
The bottom line: your team is probably getting many key assumptions wrong in your feature decisions without knowing it. The key is to accept what you can and cannot know ahead of time, to avoid compounding poor accuracy with poor processes.
Bad decisions and consequences
“What gets us into trouble is not what we don’t know. It’s what we know for sure that just ain’t so.”
Until you come to grips with this uncertainty and natural biases, there are negative consequences of overconfidence that may be hurting your product team’s performance.
Poor planning inputs: Goals are set based on overly optimistic assumptions for execution, schedules, user adoption, impact and success rates, sometimes called the “EGAP Principle” (Everything Goes According to Plan). This leads to targets frequently being missed and teams scrambling to make up deficits.
Decisions made by loudest voices: Those expressing more confidence without necessarily being more informed can be overly influential, with others on the team playing along despite their doubts; this can lead to selection of worse quality ideas.
Not accounting for risk in plans: Teams who believe v1 will meet their goals plan for bigger releases, don’t budget sufficiently for multiple experiments and assume full immediate benefit; this means they are taking away from an iterative approach most likely to achieve success.
Hesitance to report bad news. In organizations where success is assumed, it becomes uncomfortable to report feature results that miss their targets. This creates a disincentive to evaluate and report on outcomes objectively.
Staying unaware: Perhaps most dangerous is that these teams are inconsistent about measuring feature results. This means they miss out on learning, calibration and critical decisions on what to do next. Without awareness, they don’t improve. Ignorance is not bliss.
Strategies Given an Uncertain Reality
This is not meant to be a pessimistic article. Feature changes and experiments are the means to improve customer and business value. Once you accept this reality, you can evaluate and change your approach for the better.
Things that can be part of maximizing your impact in this uncertain reality:
Measure and act consistently: Evaluate feature launches consistently; report on and share results; take action based on what you have learned. This provides a feedback loop to calibrate confidence but also ensure that your decisions are based on actual data rather than overconfident assumptions.
Increase ability to detect: Test one variable at a time to avoid ambiguous results; prioritize “high contrast” experiments with greater potential for material impact and thus signal.
Reduce chance of being wrong: Conduct enough customer research to build an understanding of the problem and thus greater confidence in the solutions; consider reasons why you might be wrong as well as why you might be right to calibrate confidence.
Reduce cost of being wrong: Start with the smallest possible solution e.g. a cupcake; do more frequent, iterative tests to limit development costs; defer solving for scale until you have signal that scale will be needed; target specific audiences to contain impact; roll back tests that didn’t work to avoid a drag on user experience and complexity.
Increase reward for being right: Where you see positive results, look for analogous opportunities when the odds are in your favor: scale and expand based on wins observed in smaller tests, keep mining results that showed a positive outcome.
Run more experiments and iterate: Knowing that many tests won’t yield your desired outcome, increase the frequency of experimentation and iteration rather than aiming for perfection on each.
Improvement is a Process
The teams that embrace this uncertainty don't become paralyzed – they become more systematic about learning. They measure more, iterate faster, and ultimately deliver better outcomes because they're working with reality instead of against it.
“People who think that they know more than they do are less motivated to learn and improve than those who understand their limitations. Indeed, one study showed that the least capable people have the largest gap between what they think they can do and what they actually achieve.”
This is one area of building high-impact product teams. I’d love your feedback on what works for you.
If you’re looking for help, please reach out and we can discuss your situation.
