Not So Fast: Analytics, Disruption, and Organizational Change

Will Goodrum

September 29, 2017


A recent client of ours found themselves in a sticky situation. They are a long-established manufacturer, with one of the largest market capitalizations in their industry. For decades, they have been a trusted vendor to their customers, in a field where reliability and quality have been paramount.

But lately, their environment was changing. “Disruptors” were entering their market who could sell competing products at a lower price, with lower quality and less functionality. These disruptors seemed to have better command of digital technologies and were adept at using data. Our client had plenty of data, but was only just starting to think strategically about how this data could be used to drive future growth and market opportunity. As profits started to slip, and sales shifted toward the cheaper, newer entrants, it became imperative for the client to get a handle on this digital disruption.

Analytics: A Disruptive Force

Few business theories have been as influential and pervasive as Clayton Christensen’s disruption theory, as described in the Innovator’s Dilemma. Disruption theory explains how established firms, with expensive and complicated products are “disrupted” by new entrants whose products are not as sophisticated but are cheaper. As the new entrants’ products are adopted, and their profits grow, they further erode the leader’s dominance. Eventually, the new entrants supplant the leader as the preferred vendor in that market.  Recent articles in HBR (including one co-authored by Christensen) have highlighted that while many of the tenets of disruption theory remain pertinent in a post-industrial, post-dotcom bubble economy, the nature and velocity of the disruption is changing and accelerating.

As an example of this changing disruption, Wessel notes that new disruptors tend to be venture funded, software-oriented businesses that can (and frequently do) raise enormous amounts of capital. These new entrants are frequently asset-light by comparison to their established competitors; they borrow money on equity, not credit. Existing players find it difficult to raise and invest capital quickly enough in the technologies and strategies that they need to develop to compete effectively with the disruptors.

Despite their disadvantage in raising funds, dominant, market-leading companies do have an advantage over their newer, less-experienced competitors: a wealth of data. Established companies have vast resources of information about their customers and product offerings which, when explored with analytics, will reveal business value that disruptive entrants cannot hope to tap. Since many of these entrenched players have not viewed their data as an asset, analytics offers the promise of doing new things in new ways -- its own kind of disruption. As a result, companies are pouring money into data infrastructure, data science talent, and analytics projects to revitalize their operations, identify efficiencies, and develop new products and services. By International Data Corporation’s estimates, the market for data analytics will surpass $200 billion dollars in 2020, as companies pour more of their resources into these activities.

Analytics: As Disruptive a Force as We Think?

The trouble is, established companies view analytics as the solution to their disruption problems, rather than merely a possible advantage over their disruptive competitors. This is a subtle, but important distinction. In our more than 20 years as predictive analytics consultants, we have seen that many of the obstacles to successfully implementing analytics are the same obstacles that prevent companies from productively responding to disruption. These include:

  • Insufficient data to address the problem of interest
  • Legacy organizational structures and IT infrastructure
  • Inflexible people and processes
  • Poor communication between functional groups

A recent survey by Newvantage Partners showed how data initiatives are just beginning to address the problems companies face in disruption. As shown in Figure 1, 44.3% of companies were successful using data investments to identify new avenues for innovation. However, 35.5% of companies had not yet started to use data in this way. More troublesome was the 48.8% of respondents who said they had not yet started using data to transform their business for the future (only 48.4% said that they had).


Figure 1. Survey results of various data initiatives and their success rates (Credit: IDC)

So, a company’s success with analytics is subject to many of the same obstacles and constraints that make those firms susceptible to disruption, and few firms have successfully employed analytics to transform their business for future operations.

Using Analytics as a Response to Disruption

Even though analytics is not inherently a solution to the challenges of disruption, there are proven ways to ensure success with analytics that should be a part of any plan to respond to disruption.

  • Develop a Forward-Looking Analytics Strategy: It is a very difficult for most businesses to imagine a new way of doing things. However, that is precisely the response that disruption demands. As Martin and Golsby-Smith wrote, it is necessary to break free of the “status quo trap;” the belief that things will only be as they are. Analytics offers the potential to make decisions in new ways, but it is grounded in that it is based on historical evidence. Change is not merely doing old things better. If new insights are to be distilled from the data, then an analytics strategy should be constructed and executed in a way that will enable success. This means prioritizing prototyping, collecting or acquiring new data sources, and reorganizing people and processes to meet these challenges.
  • Emphasize Quick Wins: Nothing breeds excitement as easily as success. While it may be tempting to address disruption with an analytics “moonshot,” a recent McKinsey study revealed that smaller initiatives are “often easier and quicker to execute: their small size involves fewer layers of approval and less coordination.” They split initiatives by size, or project value, into boulders (> 5% pipeline) pebbles (0.5%-5%), and sand (<0.5%).  The authors found the recurring impact of the pebbles and sand initiatives accounted for 50% of the realized value on average (see Figure 2). Getting early wins, and celebrating them will breed excitement and drive change.

Figure 2. Recurring impact of a project based on size (% of total value)

  • Foster a Culture of Open Communication and Problem Solving: Doing new things will inevitably lead to more frequent mistakes or missteps. If you are the established leader of your market, then your processes and structures are likely optimized for your current solutions. It can be difficult to migrate from a culture of efficiency to one of agility. However, our most successful clients have fostered cultures of open communication that are focused on problem solving (not problem shaming). We speak from our own experience, as well. At Elder Research, many of the initial or intermediate hypotheses that we try when helping our clients do not pan out, but those failed first forays are necessary to refine and shape our valuable final deliverables.

For example, our ongoing engagement with a major hospital system continues to move from success to success, thanks to the openness and respect that pervades the organization from top-to-bottom. Frontline staff are comfortable expressing their opinions to their leaders, and do so with thoughtfulness and courtesy. Similarly, management clearly articulates their decision making, which engenders confidence.


How did our client respond to their disruptors? They began by building a robust data collection pipeline and infrastructure for data storage. Data governance and standardization were given early emphasis, since the analytical models rely on high-quality data. Lastly, they identified important products in their portfolio that could benefit from predictive modeling, and began developing and deploying models into production. As an agile strategy, this is a continuously evolving process, and modifications are under consideration. As results are starting to come in, they are quickly learning and growing as they disrupt themselves.

When facing the new challenges posed by a disruptive competitor, it is sensible to turn to analytics to find a new way of doing business. However, analytics on its own is not a solution to the problems posed by disruption. In fact, many of the challenges that make responding to disruption difficult are the same ones that derail analytics projects and prevent them from fully maturing! Yet, established players can make good use of analytics as part of a strategic response to disruption. They must develop a forward-looking analytics strategy, emphasize quick wins to build momentum and stakeholder support, and foster a culture of open communication and problem solving.  Then, established leaders can get the data-driven insights they need to stem the whelming tides of disruption in their field.

Request a consultation to discuss how Elder Research can help you use analytics to disrupt your competitors.


Read the blog Top 3 Keys to Leading a Successful Data Analytics Initiative

Read the blog Developing an Effective Analytics Strategy: A Roadmap for Success

Read the blog Avoiding the Most Pernicious Prediction Pitfall

About the Author

Will Goodrum Data Scientist Will Goodrum has a decade of experience applying numerical analysis and engineering to solve practical problems and generate value for customers. Previously, Dr. Goodrum worked in an engineering software firm, helping medium-to-large scale customers across industrial sectors develop superior products and reduce their time-to-market. As a graduate student, he applied statistical modeling and physics-based simulation to estimate the impact of policy decisions on lifetime maintenance costs for a regional transportation authority. Will holds a B.S. in Mechanical Engineering from the University of Virginia, and a PhD in Engineering from Cambridge University.