The Modern-Day AI Executive: Most AI Investments Return Zero

Perhaps one of the most difficult facts to believe in the field of AI and data science is that most investments in these projects fail to return meaningful value to the organization. Indeed, multiple published studies indicate that over 80% of these projects fail.

Imagine 80-90% of software engineering projects failing to meet user needs! Or 80-90% of oil and gas extraction efforts yielding no financial return. Even with the current advanced level of analytics talent, tools, and understanding, why is its failure rate so high? These numbers, if sustained, would typically signal a struggling, if not dying, industry.

Are AI initiatives especially challenging?

What makes their failure rate so high compared to other business undertakings?

In ample writing about these failures fingers have been pointed in many directions. Elder Research has over 25 years of experience within this space, and a successful implementation rate of about 90% (hard won from those many years). Perhaps we can help mitigate risks in your organization?

In our experience, there are two primary causes we have seen that lead to data science and AI project failure:

AI is, and continues to be, a new discipline that is still uniquely foreign to most enterprises.

Organizations must continuously innovate and transform themselves if they are to maintain relevance. But innovation and transformation can happen in two largely different dimensions.

Dimension: Existing Processes & Technologies

One dimension represents existing processes and technologies. For instance:

• the customer interaction is going digital (i.e., a web-based system instead of in-person and paper)
• the manufacturing line is gaining efficiency (i.e., lean principles for process improvement and waste reduction)
• the finance and accounting process is being streamlined (i.e., consolidating and integrating systems)

Dimension: Innovation & Transformation

AI, and its neighbors data science and machine learning, live in the other dimension of innovation and transformation that is much more radical and disruptive because it:

• requires new talent (i.e., data scientists, machine learning engineers, chief data officers)
• introduces new processes (i.e., data governance, model development, model governance)
• creates and deploys new technologies (i.e., enterprise data storage, machine learning, natural language processing)

AI as a Change Agent

These new “AI” people, processes, and technologies did not substantively exist to impose themselves upon organizations a few years ago. So it is understandable that most organizations and enterprises see this movement as largely foreign and frankly strange and difficult to understand.

Every organization has a pre-existing “way” of doing business, and AI is a foreign actor in the system; it appears to roam around like a rogue agent, asking for things that don’t normally get asked for:

• You want all the data across the entire organization?
• You want how much compute?
• And for what?

Likewise, producing products, services, and solutions that people don’t normally know how to use results in questions such as:

• You’re telling me the data is saying the opposite of what I’ve done for the last 20 years?
• You want to automate this aspect of the business?
• You want me to make an important business decision using this machine learning model instead of my expert judgment?

An organization’s ability to “absorb” this change agent (not rogue agent) is a big leading indicator of long-term AI success. Or to say it conversely, an organization’s inability to “absorb” this change agent is a big leading indicator of a slow and painful AI death.

AI initiatives embody more uncertainty (risk) than most projects because of the data!

Of course, every new business endeavor, or investment, has risk. Perhaps a new product fails to gain market acceptance, macro-economic trends shift unexpectedly, a pandemic closes down entire sectors of the economy, or poor market research leads to services without a buyer — the list goes on. These are risks organizations regularly contend with and seek to mitigate.

Most would agree the bigger the risk, the bigger the reward. At Elder Research we believe projects in the data science and AI space embody a greater risk of failure (as is indicated by the statistics cited earlier) than projects in other domains yet tend to yield greater results when implemented successfully. Why do AI/DS projects embody greater risks? Because of the data!

The Building Blocks of AI

AI (and data science and machine learning) projects are built upon a raw material (data) that is often captured, or designed, without any automated learning in mind. This is particularly true for legacy organizations that, for decades, have collected data for operations, without deep thought about its strategic value. Of course, that is changing now as leaders have come to recognize the strategic value good data can provide. However, the process of extracting that value, either through big or small projects, is one that contains significant risk because the starting point, the raw material, is initially opaque and requires analysis before meaningful AI development and data science can begin.

Data is usually of much lower quality than expected:

• much of it may be missing,
• in the wrong format,
• collected poorly or in a biased fashion,
• be un-representative of you want to measure,
• is not understood, be confusing or misunderstood,
• or not have the appropriate infrastructure to support analysis, development, deployment, and monitoring, etc.

These obstacles are not trivial and are usually quite challenging to overcome.

The Challenging Side of AI

Moreover, data shortcomings are not uncovered until after a project is launched. AI and data science projects involve a significant discovery process, one that can require investment in exploratory analysis before a complete understanding of what is feasible materializes. These uncovered flaws cause timelines to falter and necessitate additional investments in tools, processes, infrastructure, and development to overcome.

These after-the-start discoveries can lead to discouraging initial results which dampens enthusiasm among the business leaders who sponsor the projects. As blame circulates as to why these stumbling blocks appear it is important to remember that a fundamental ingredient in these projects is data, and it is not until we are long into the initial discovery phase until the data is sufficiently understood.

Lack of awareness of this fundamental dependency of data science and AI projects will inevitably lead to disappointment unless expectations are managed. Unfortunately, data science and AI projects tend to get lumped in together with their close cousins in software engineering and IT projects. Yet, the latter type of projects are more deterministic (think of building a wall brick by brick) and their raw material is much better known from the beginning. Data science and AI instead begin with an inherited raw material (data) that defines the starting point.

Artificial Intelligence is a tool for Data Scientist

Realizing that data science and AI projects operate differently than most other technical projects means they must be treated differently. It is important to buffer project deadlines to account for the inevitable discoveries – good or bad – that will take place. That will help expectations to be calibrated appropriately to avoid the death spiral of waning stakeholder buy-in and executive sponsorship.

With over 25 years of consulting experience in this space and having served hundreds of clients in almost every sector, we have seen many client stories play out. Some tragedies, some horror novels, many bumps and bruises, but also many stories that embody heroics and courage that produce success and astronomical value.

We drew on our experience to create the following diagram on the ingredients necessary to foster successful AI and data science projects within your organization:

Each element in graphic above represents a large topic on its own, but the key idea here is that organizations and enterprises must embody and nurture all of these components working in unison for long-term success. Much like a dogsled team requires every dog to run in the same direction at the same time, AI and data science initiatives require each element to ‘run’ in the same direction, at the same time, and in unison.

As long-term practitioners, we have grappled with the complexities involved with implementing AI within many types of organizations. Often, the initiatives struggled to gain traction in one or more of the elements in the graphic above until it was diagnosed and competencies could be improved or brought to bear. An organization must develop and harness culture, processes, and governance that understands, respects, and maintains a strong practice in each area. Doing so will tremendously reduce the risk of failure.

For success, AI initiatives require tightly coupled interactions between each of these elements. The project team, led by a product or project manager, and the technical staff, rely heavily on each organizational function as they move through the stages of ideation, prototype development, model development, model validation, implementation, and follow-on monitoring and maintenance. Organizations must create culture, processes, and governance to enable seamless interactions between these elements. AI success is always a team sport!

AI initiatives are demanding. It is one thing to aggregate, clean, model, and make inferences about the future with data in a controlled experiment. It is something else entirely to seamlessly develop, deploy, integrate, and operate AI across an organization so that it augments humans on a consistent, viable basis.  But it is doable, and the rewards of success are exponential!

Footnote:

AI is new in that it is built upon the intersection of many decades-old and robust sub-disciplines like mathematics, statistics, engineering, data analysis, computer science, operations research, and product development. But AI brings together, fuses, and deeply integrates these sub-disciplines in ways that haven’t been done before (or weren’t possible before).