The Innovation Challenge: Good Ideas Are Harder to Find
Innovation has been the driver of the extraordinary progress from which humankind has benefited for a couple of centuries, but it faces a largely hidden threat: Innovation is becoming harder and more expensive.
Innovation is an enabler of human progress
It’s instructive here to take the long view. For most of recorded human history, improvements in human welfare from generation to generation have been limited. Take, for example, GDP per capita as a measure of economic prosperity. For most of human history, roughly until the early 1800s, the measure barely moved to $1,200. But since that time, it has grown by more than 14 times. Human health has followed a similar trajectory—low for centuries and only significantly improving in recent generations. In 1900, for example, the average life expectancy of a newborn was 32 years. By 2021, this had more than doubled to 71 years.
These and many other improvements in our lives have been driven by a set of scientific discoveries and products engineered based on those breakthroughs. These innovations have enabled economies to grow and people’s lives to improve. The steam engine helped power the Industrial Revolution. Vaccines that prevent diseases such as smallpox, measles, and polio continue to save millions of lives each year; infant mortality is estimated to have decreased 40 percent in the past 50 years because of vaccines. The invention of the integrated circuit for computing and lasers for communication through fiber-optic cables helped create the global internet.
But the rate of progress enabled by innovation now faces an under-recognized threat: Innovation is getting more difficult and more expensive.
Even as science advances, R&D productivity is on the wane
By many metrics, and in many fields, each dollar spent on R&D has been buying less innovation over time. In other words, R&D productivity has been declining.
Take the semiconductor industry. With integrated circuits embedded in products that support nearly every part of our lives, this sector has advanced in accordance with “Moore’s Law”—the remarkable observation put forward by Intel cofounder Gordon Moore that the number of transistors on an integrated circuit will double about every two years. This is roughly equivalent to an exponential growth rate of 35 percent annually in transistors per dollar.
But this level of performance increase has been bought at the cost of increasing expenditures in R&D. Nicholas Bloom, an economics professor at Stanford University, and his research collaborators published a paper in 2020 that examined the real R&D expenditures of semiconductor companies and equipment manufacturers and estimated that their annual research effort rose by a factor of 18 between 1971 and 2014. In other words, maintaining the performance growth rate in Moore’s Law required 18 times more inflation-adjusted R&D spending in 2014 than it did in 1971.
It’s not just semiconductors. The biopharmaceutical industry has produced innovative products used to prevent and treat many diseases, enabling millions of people to live longer and healthier lives. But the challenge of declining R&D productivity in that industry led Jack Scannell, a multidisciplinary life sciences analyst, researcher, and entrepreneur, to coin the term “Eroom’s Law” (that is, the reverse of Moore’s Law) to describe the fact that drug discovery has become slower and more expensive over time. He and his research collaborators found that the number of new drugs approved per billion US dollars spent on R&D halved roughly every nine years between 1950 and 2011, falling around 80-fold in inflation-adjusted terms.
Declining R&D productivity has been reported in other fields, such as agriculture, where higher yields for multiple crop types require increasing levels of R&D spend. Using company-level data across all sectors in the United States, Bloom and his team found that R&D productivity declined in general, with output measures including revenue, market capitalization, employment, and revenue per employee.
AI has the potential to bend the curves of R&D productivity, not only unlocking more economic growth but also boosting the chances of solving some of the most important human challenges, from preventing and curing diseases to reducing the level of carbon emissions.
Over the past decade, we have seen how AI, when coupled with complementary management practices to rewire the way organizations operate, can generate real business value. Even prior to the advent of gen AI, analytical AI was being used by roughly half of the enterprises represented in McKinsey’s Global Survey on AI. Those organizations have been deploying the technology across a variety of business functions—from increasing revenue through more targeted marketing to reducing costs in supply chain operations. Since ChatGPT became available in late 2023, the percentage of organizations reporting that they use AI has spiked upward by 20 percent, with companies implementing gen AI in use cases from customer service to software engineering.
Most of these applications of AI have been aimed at improving the efficiency of existing tasks and workflows. But boosting efficiency and productivity is just one way that AI promises to unlock a new era of growth and opportunity. Our research shows that AI also can be deployed to accelerate innovation to create entirely new products and services. To put it another way: AI can be used to bend the curves of the declining R&D productivity we documented in the previous section.
We have identified three primary channels through which AI technologies can accelerate innovation, each with a corresponding type of model: increasing the velocity, volume, and variety of design candidate generation; accelerating the evaluation of candidates through AI proxy models; and accelerating research operations.
Increasing the velocity, volume, and variety of design candidate generation
A simplified model of the R&D process consists of identifying a set of customer needs, generating candidate designs, and then evaluating those ideas to identify the most promising ones that will best meet the needs of the customer or user. One of the highest potential opportunities for AI to enhance innovation is to more quickly generate a greater volume and variety of design candidates.
Gen AI technology is based on foundation models—very large simulated neural networks that are trained on vast collections of data to take unstructured data (that is, data that isn’t best stored in rows and columns like a spreadsheet, such as human language) as inputs and then generate unstructured data as output. Large language models (LLMs) are the best-known types of foundation models, underpinning the chatbots that have made gen AI such a compelling technology.
FAQs
Q: How can AI accelerate innovation in R&D processes?
A: AI can accelerate innovation by increasing the velocity, volume, and variety of design candidate generation, accelerating the evaluation of candidates through AI proxy models, and streamlining research operations.
Q: What industries can benefit the most from AI in R&D?
A: Industries such as computer gaming, pharmaceuticals, electronics, and consumer goods can benefit significantly from using AI to accelerate their R&D processes.
Q: What are the key organizational shifts required to leverage AI in R&D?
A: Organizations need to move quickly and scale rapidly, rewire their organizational structures, build core competencies around AI models, and be thoughtful about incorporating humans in the loop in R&D processes.
Conclusion
In an economy driven by innovation, the ability to generate new ideas and products is paramount. AI technologies offer a unique opportunity to accelerate the R&D process, leading to increased productivity, efficiency, and the potential for groundbreaking innovations. By leveraging AI to generate design candidates, evaluate candidates through surrogate models, and streamline research operations, organizations can unlock new pathways for growth and prosperity.
However, realizing the full potential of AI in R&D requires strategic organizational shifts, including moving quickly, rewiring structures, building core competencies, and thoughtful integration of humans in the process. By embracing this new era of innovation and acting decisively, companies can stay ahead of the curve and drive meaningful progress in the years to come.
