Blog

5 Key Reasons Why Analytics Projects Fail

Peter Bruce

February 22, 2019

BLOG_5 Key Reasons Why Analytics Projects Fail

With the news full of so many successes in the fields of analytics, machine learning and artificial intelligence, it is easy to lose sight of the high failure rate of analytics projects.  McKinsey just came out with a report that only 8% of big companies (revenue > $ 1 billion) have successfully scaled and integrated analytics throughout the organization. In some ways, the very notable successes of analytics and data science contribute to the high failure rate, as ill-prepared organizations flock to implement projects.  There are various reasons for failure, and all are instructive.

A respected voice in data science, John Elder, shared his experience a number of years ago with a popular lecture called “The Top Ten Data Mining Mistakes.”  When it first came out, technical statistical errors ranked high on the list - mistakes like overfitting the data used to train a model, ignoring outliers, failing to try multiple methods, and accepting leaks from the future.  As time passed and analytics projects became widespread in many different organizations, attention has shifted to the organizational climate surrounding the implementation of these projects.

Here is a list of five principal reasons organizational projects fail:

1.  Shiny Toy Syndrome  

The “shiny toy syndrome” occurs when top leadership sees other organizations implementing analytics and data science projects, and wants a project of their own.  This leads to a number of problems:

  • Top management’s attraction is to the shininess of the toy, not to a broader analytics strategy.

  • A clear cut problem to solve, and analytics objective, are typically missing.

  • The rest of the organization sees that top management’s attention is shallow, and not likely to last long enough to make sustained coordination and effort worthwhile.

2.  Focus on Vendors’ Tools - The “Plug-and-Play” Illusion

Analytics is known to require sophisticated software and algorithms, so vendors with these tools find a ready reception.  The problems?

  • The software vendor is not usually in a position to define the organizational problem.

  • The result is a tool in search of a problem.

  • A further outcome is disillusionment with analytics, since no organizational problems were solved and the organization is left with an expensive but unused tool.

3.  Rely Solely on Specialized Team of Super-experts

It is well known that successful deployment of analytics requires expertise in the disciplines that make up the field of data science - statistics, computer science, IT.  The problem comes when a company assembles such a team, but fails to integrate it with the rest of the company.

  • Strategic direction from the top will be lacking, and there is no assurance that important problems will be worked on.

  • The team’s elegant and technically advanced models do no good if they don’t deliver solutions to problems that need to be solved, in harness with the rest of the organization.

4.  Fail to Spread Analytics Education Broadly

If analytics is to escape its silo and become a strategic focus of the organization, some basic analytics education must be spread broadly throughout the organization.  If the various functional units do not know what analytics and data science can do, it is difficult to get everybody pulling in the same direction.

  • Strategic leaders, not really understanding analytic concepts and details, will fail to identify and frame useful problems.

  • Data gatekeepers may impede access to data.

  • Useful knowledge about the data from functional and domain experts may remain unexplored.

  • Those at the deployment end may not trust the analytics, and carry on with the old ways.

5.  Technical Reasons

The technical errors that John Elder identified over a decade ago in his “Top 10 Mistakes” lecture remain common.  One is overfitting the data, or “fitting the noise, not the signal.” A related error is running huge numbers of model variations, and believing the best model.  In both cases, some element of random chance is being misinterpreted as something interesting (a common phenomenon - there is a human predilection to latch on to seemingly meaningful chance patterns).  Other technical errors can actually be mitigated by avoiding the first four errors mentioned above. For example:

  • Mistaken inclusion of future information in training a prototype model can be avoided if domain experts are better trained in analytics and included in the process.

  • Unwarranted extrapolation beyond the range of a model can be avoided in the same way.

  • Technical experts often turn a poorly specified scenario into a problem they can solve, and yet the solution has little meaningful utility; this can be avoided if top decision-makers who set strategy are better versed in analytics.


Download "The 10 Most Common Data Mining Business Mistakes" by Elder Research COO, Jeff Deal, to learn more about the practical application of analytics to your business problems.

Want to dive deeper yourself?  Statistics.com offers dozens of fully online courses - learn more


Related

Top 3 Objectives Before Starting an Analytics Project

Why Data Literacy in the C-Suite Matters

Hiring a Data Analytics Consultant

Building a High-Functioning Analytics Team


About the Author

Peter Bruce Peter Bruce is Founder and President of The Institute for Statistics Education at Statistics.com. Previously he taught statistics at the University of Maryland, and served in the U.S. Foreign Service. He is a co-author of Data Mining for Business Analytics, with Galit Shmueli and Nitin R. Patel (Wiley, 3rd ed. 2016; also JMP version 2017, R version 2018, Python version 2019; plus translations into Korean and Chinese), Introductory Statistics and Analytics (Wiley, 2015), and Practical Statistics for Data Scientists, with Andrew Bruce, (O'Reilly 2016). His blogs on statistics are featured regularly in Scientific American online.