Why do the vast majority of innovations fall so far short of ambitions? Companies have never had more sophisticated tools and techniques in their arsenal. More resources than ever – including billions of dollars and droves of data – are deployed in support of reaching innovation goals. We’ve never known our customers better. But innovation is still hit or miss. Clayton Christensen, the world’s foremost authority on innovation and growth, believes the fundamental problem is that we’re not asking the right questions.

Most innovators focus on “How can we make our products better, more profitable, or differentiate them from the competition?” Few wonder why I might purchase their product, he says. “After decades of watching great companies fail over and over again, I’ve come to the conclusion that there is, indeed, a better question to ask: what job did you hire that product to do?”

Simply put: understanding customers does not drive innovation success. You have to understand their jobs to be done.

This core insight first emerged in a course Christensen teaches at Harvard Business School and introduced in his 2003 book, “The Innovator’s Solution.” In the years since, the Jobs to be Done theory has been refined, informed and influenced by his numerous conversations with colleagues, collaborators and other thought leaders. It’s been validated and proven in the work of some of the world’s most respected business leaders and innovators – Amazon’s Jeff Bezos and Intuit’s Scott Cook to name just two – as well as in the founding of highly successful entrepreneurial ventures in recent years. Think Airbnb, Khan Academy and Chobani yogurt.

But in hearing an executive refer to Jobs to Be Done theory as “milkshake marketing” – an affectionate nod to one of Christensen’s oldest and favorite stories – he knew a deeper, more definitive and meaningful treatment of the theory was needed to “save it from being turned into one of the many ‘easy fixes’ that flood the marketplace of business ideas.” It’s a completely different lens on innovation with terrific potential to reframe industries. And it’s the subject of his forthcoming book, “Competing Against Luck: The Story of Innovation and Customer Choice” (HarperCollins, October 2016).

A generation ago, Christensen revolutionized business with his groundbreaking theory of disruptive innovation, a model of competitive response. But, he acknowledges, disruptive innovation doesn’t tell you where to look for new opportunities. It doesn’t predict or explain how, specifically, a company should innovate to undermine the established leaders or where to create new markets. Jobs to be Done – and “Competing Against Luck” – do.

Innovation may never be a perfect science, but that’s not the point, argues Christensen. “We have the ability to make innovation a reliable engine for growth, and one based on a clear understanding of causality, rather than simply casting seeds in the hopes of one day harvesting some fruit. But it does require asking different questions and thinking in new ways.”