The Hidden Reason Innovation Failure Rates Have Not Improved in 30 Years
Retired venture investor Jerry Neumann recently published an essay in the online publication Colossus titled “We Have Learned Nothing.” His argument is direct: after thirty years of startup methodology, failure rates have not improved. Customer development, Design Thinking, the Business Model Canvas, Lean Startup, and Effectuation were all named as methods taught and widely adopted. All taught in accelerators, universities, and incubators around the world. And yet the data has not moved. Failure rates remain high.
The observation extends well beyond startups. Corporate innovation and product teams have adopted the same methods and inherited the same assumptions. The failure rates have not improved there either.
It is a well-reasoned piece, and it deserves a serious response. Neumann offers three possible explanations. Maybe the theories are simply wrong. Maybe they were so obvious that formalizing them was pointless. Or maybe, once everybody adopted the same playbook, the methods stopped conferring an advantage. He leans toward the third explanation, a “Red Queen” dynamic in which competitive convergence erases whatever edge the methods once provided.
I would like to offer a fourth perspective. It comes from twenty years of watching talented, well-resourced teams do everything right and still fail, and from spending seven of those years discovering why.
The methods Neumann mentions were all designed to mitigate the risk of failure. But they failed to distinguish the difference between market risk and solution risk. They have been effective at minimizing solution risk, the risk that the team cannot build, deliver, or scale the product. But they are inherently limited in their ability to mitigate market risk, the risk of not understanding the customers’ unmet needs so they do not buy what you make. For innovation to become a repeatable business process, it must be executed by discovering the target customers’ unmet needs first, and only then devising solution ideas to address the selected unmet needs (opportunities).
The False Belief That Shaped the Toolkit
Every methodology Neumann names shares a false belief: customers cannot tell you what they want. The belief has been bolstered by a famous quote attributed to Henry Ford: “If I had asked customers what they wanted, they would have told me a faster horse.” The quote is almost certainly apocryphal, but it persists because it resonates with a real challenge: getting reliable customer input is difficult if you do not know what types of inputs to obtain.
From this belief, the founders of these methods drew a reasonable conclusion: if customers cannot articulate their needs, then the most productive thing a team can do is generate an idea, build something minimal, and test it. Lean Startup begins with a hypothesis and tests it through a minimum viable product. Customer Development begins with assumptions about customer problems and validates them through interviews. The Business Model Canvas begins with a value proposition and maps the business around it. Design Thinking begins with empathy for the user and moves quickly to ideation and prototyping. Effectuation begins with the founder’s own skills and network and builds outward from there.
All five start after assumptions are formed, or bypass the question of customer needs entirely, because their creators concluded there was little systematic to do before that point. It is an understandable conclusion. I held the same assumption early in my career. Most of us did. The shift came not from theory but from watching what happened when teams asked customers different questions.
What misled people is the fact that this statement, “Customers cannot tell us what they want” can be interpreted in two ways. More specifically, the phrase “what they want” has two completely different meanings. It is true that customers cannot reliably tell us what solutions or featuresthey want. That requires expertise in the solution domain (e.g., technology, engineering, etc.) they do not possess.
But customers can tell us what they want to accomplish: their jobs and desired outcomes. These are need statements that contain no reference to any product or service solution. That taps into customers’ lived experience, about which they are the authority. When we ask what they are trying to accomplish and where they struggle, we access expertise they have and can articulate.
Consider a physician who fails to ask a critical diagnostic question during a patient assessment. The patient does not offer the information. The condition is misdiagnosed, and treatment fails. Was that failure due to the patient’s inability to describe symptoms, or the physician’s failure to ask the right question? The same dynamic applies to innovation. When customers struggle to articulate needs, the failure is in our questioning, not in their ability to articulate. “Latent unarticulated needs” remain hidden until a JTBD practitioner asks the right questions.
The Ford quote answers the wrong question. “What do you want?” invites product specifications. Customers cannot reliably provide those. Ford knew the job to be done was to travel from A to B. Then the question becomes “How will you know that you have been successful?” and “Where do you struggle?” Customers can and will tell us this information if we know what to ask. In fact, customers will give you instructions about where to focus and what to do to create new value. Most teams do not believe this is even possible, let alone know how to do it.
The CB Insights data confirms where innovation risk actually lives. Forty-three percent of failed startups cited poor product-market fit, a figure that has remained effectively unchanged across the entire period of methodological adoption. Running out of capital was cited more often, at 70%, but the researchers note that it is almost always the final cause of death rather than the root cause. The root cause, in nearly half of all cases, is that the team built something the market did not need.
That is not a solution risk problem. That is a market risk problem. And no amount of faster iteration, leaner MVPs, or better-structured experiments will address it, because all of those tools engage after the foundational assumptions have already been set. Even Steve Blank, the architect of the Customer Development method, acknowledged in 2021 that his tools lack a front-end step and do not help teams figure out where to start the search.
The Work That Must Come First
The innovation industry has long recognized the work that precedes ideation. It has been called the “fuzzy front end.” But the label reveals the invalid assumption. The front end was called fuzzy because practitioners believed the uncertainty was inherent to innovation itself. If customers cannot tell you what they need, then of course the earliest decisions feel unstructured and risky.
But the front end of innovation is not inherently fuzzy or risky; the way most companies execute it is: ideas-first. Between recognizing the need to innovate and committing resources to create an MVP, teams must determine the following (if you want to turn innovation into a repeatable business process):
1. Determine the business objective for the innovation effort
2. Identify the target customer for whom you want to create value
3. Discover the core job(s) they are trying to accomplish
4. Discover where they struggle, i.e., their important unsatisfied desired outcomes
These are the highest-consequence decisions in the entire innovation cycle. They are the information that makes confident innovation decisions possible. In most organizations, they are made on assumptions rather than with precise processes and definitions. Not because people lack talent or effort, but because the industry has never built systematic infrastructure for this work. We have infrastructure for ideation, for testing, for iteration, development, and delivery. But the work that precedes all of them has been left to intuition, pattern recognition, and internal consensus.
None of this requires exotic expertise or years of preparation. It requires knowing which questions to ask, in what sequence, and what to do with the answers. Teams that learn this discipline typically begin generating actionable customer insights immediately.
Consider what happens in practice. A team identifies what they think is a market opportunity. They generate ideas. They form hypotheses about what customers want. They build a minimum viable product. They test it. If it fails, they iterate. If it fails again, they pivot.
Often, this process proceeds without ever talking with customers to reveal their jobs and desired outcomes. You do not need an MVP to discover what customers are trying to accomplish and where they struggle when you know what type of inputs you need. In fact, introducing an MVP before discovering the customers’ unmet needs can inject social desirability bias (error) into your customer conversations.
In most companies, hypotheses are assumptions from the start. Iteration refines the solution, but it cannot establish product-market fit that was never grounded in validated customer needs. The team is searching for that fit through repeated experiments, each one consuming time, capital, and organizational attention.
Every one of these activities is designed to mitigate solution risk. None of them mitigate market risk. And market risk is where most failures originate. If you want to turn innovation into a repeatable business process, the sequence is not optional. Discover where customers struggle first. Devise solutions to address the most attractive opportunities second.
Methodologies designed for this work do exist. Jobs-to-be-Done theory and Outcome-Driven Innovation® were built specifically to reveal customer needs before ideas are generated. But ODI’s comprehensive quantitative rigor requires significant investment in research infrastructure, which has limited its adoption.
Additionally, the same false belief that shaped the mainstream toolkit limited the ecosystem’s receptivity to any methodology built on the premise that customers can articulate their needs. The startup ecosystem, the accelerators, the university programs, and the corporate innovation playbooks still overwhelmingly begin where Lean Startup begins: with an idea and a hypothesis.
I created Lean JTBD OS™ to close that gap. It is built on the language of Outcome-Driven Innovation® with the permission of Tony Ulwick, and it is designed to be as simple as possible and only as complex as necessary. Mastering the front end of innovation should be accessible to any team willing to learn some new things. This process precedes and complements the methods referenced and makes them more effective by providing them with the essential customer insights they need to succeed.
The purpose of this article is not to promote a specific methodology. It is to name a gap the innovation ecosystem has left unaddressed for thirty years, and to show that the work required to close it is known.
What Changes When the Front End Is Addressed
When teams discover which customer needs are unmet before they generate ideas, in a format that is separate and distinct from any product or service solutions, the team can evaluate the market opportunities and select which to pursue for new value creation. Leaders who bring this discipline into their organizations do not just reduce failure rates. They change the conversation about innovation from “let’s try and see” to “here’s why this will win.”
Instead of searching for product-market fit through repeated experiments, the team can engineer product-market fit at concept creation. The target customer’s unmet needs have been revealed. The highest-value opportunities have been identified. The solution concept was devised to address the most promising opportunities, not against an assumption. In most cases, the team does not lack creativity or technical expertise. What they lack is clarity about what to solve. Once that clarity is established, excellent ideas naturally come forward.
Fewer iterations. Faster time to proven product-market fit. Higher success rates. Not because the team got lucky, but because the leading cause of failure, misunderstanding what customers need, was addressed before resources were committed.
This is not a marginal improvement to the existing process. It changes what iteration is for. It moves product-market fit from something teams discover through trial and error to something that can be engineered at the concept stage.
Why This Matters Now More Than Ever
AI has made it faster and easier to build than at any point in history. Teams can ship prototypes in days that once took months. Code that once required a full engineering team can now be generated in hours. The cost of building a minimum viable product is approaching zero.
This is genuine progress. AI is an extraordinary tool for accelerating execution.
But it has not made it easier to know what is worth building. Every AI-powered acceleration assumes the team already knows which problems are worth solving and which opportunities to pursue. Without validated customer evidence upstream, AI enables teams to iterate against assumptions at machine speed. That is not less risky. It is more risky, faster.
The gap between execution speed and discovery infrastructure is the defining innovation risk of this moment. And it is widening.
A Different Explanation
Neumann’s Red Queen theory is elegant but it does not identify the real problem. The failure rates have not stayed flat because everyone adopted the same methods and the competitive advantage disappeared. They have stayed flat because every method mentioned skips a rigorous customer discovery that ensures only good ideas go into the pipeline for development. (A “good” idea is defined as one that targets a validated unmet need and is attractive to the firm to pursue for new value creation).
Add a hundred variations of Build-Measure-Learn and you still have a hundred teams iterating against assumptions. The front end is where the leverage is. It always has been. It is the work most methodologies skip, not because they are flawed, but because a false belief made it appear impossible and unworthy.
This is not an argument against Lean Startup, customer development, Design Thinking, or any other methodology. These are valuable approaches for refining solutions that already target a validated unmet need. We have been asking them to do work they were never designed to do. The conversation Neumann has started points toward something the innovation ecosystem has not yet tried: building systematic infrastructure for the front end of innovation. The methods are not the problem. The missing infrastructure that must precede these methods is. Without a systematic process for discovering customers’ unmet needs before generating ideas, failure rates will remain high.
The organizations that build this infrastructure first will not just make better innovation decisions. They will set the standard for how innovation is practiced in their industry.
If this resonates with you, I am putting the final touches on a companion article, ‘Why Brilliant Teams Build Products Nobody Wants to Buy (And What the Best Innovation Teams Do Differently).” Reply to me at [email protected] if you want to receive it when it is published.
