Data-driven decision making: Closing the skills gap

The essential consideration for an organization is to make sure that the data they are collecting actually address the core challenges they are trying to solve. Firm strategy and objectives should drive the entire process: from data collection, to analysis, interpretation and communication—not the other way around. As our daily lives become more digital, people are generating all kinds of ‘digital exhaust’—data that is accessible and can easily be looked at.  The question becomes, should we look at it? Is it the right data? As data grows more and more ubiquitous, it is going to become ever more important for organizations to have a strong process in place for collecting and analyzing it. Simply combing through whatever data comes easily to hand, fishing for insights, is not going to work.  There will be too much too data and too many insights.  It is essential for firms to step back and lead with the questions; and then, not be data-driven, but truth-driven. 

In his work on data-driven decision making, former Harvard professor Gary Loveman points to the importance of using data to get to the truth by setting up our questions in such a way that we can test them with data—so that we’re not just following whatever falls out of some analytic model or algorithm, but that everything comes back to an underlying truth that is important for the firm. Ultimately, data never ‘speak for themselves’—they must always come through a human interpreter. 

In your research, you found that the collection and use of data in firms is linked to performance. How so?

We were able to use U.S. Census Bureau data on roughly 35,000 plants that were carefully selected to represent the entire American manufacturing sector. The differences among plant managers in how they used data surprised me; a large number of them said they don’t feel they have enough data, or that the data isn’t driving—or even informing—their decisions. It was interesting to see how many are still quite far behind the frontier.  

This should probably not be too surprising. We know from studying other new technologies that adoption takes time. Among other things, it requires learning, and we found that the organizations that are focused on learning about the latest management practices were also more likely to adopt robust data practices. At the same time, we found that those who use data intensively —alongside other good practices and investments in information technology—had significantly higher productivity than similar organizations. Given the extent of this relationship, I expect many firms will learn about this quite quickly and make changes to their practices.

To get a sense of how big a deal this is, consider that adopting data-centric practices was associated with roughly the same productivity improvements as making very large tangible investments in information technology. In the manufacturing realm, even very small productivity improvements can be a really big deal. We were able to confirm that there is definitely something to this ‘data craze’.  Future research will be aimed at tracking this transformation and how it relates to firm performance moving forward. 

Q. You also found that the decision-making environment at a firm plays a role in the effectiveness of data-driven decision making (DDD). Please explain.

We’re still digging into this, but what we found surprised us: in general, we noted a change over the five years in our data set, in terms of how daily tasks are organized in these manufacturing plants. The overall trend is definitely towards more collaborative decision-making between managers and front-line workers; and yet, for organizations that embrace these practices, the benefits of being data-focused evaporated. So, it might be the case that, in firms where there is a good deal of autonomy for front-line workers,  embracing data-driven decision making is not as important as it is in a more traditional manufacturing setting.

My sense is that this is related to the broader context in which the data-driven practices are deployed. A firm’s culture is not something we can pin down easily in our data, but we believe it matters. When firms give workers a lot of latitude and rely less on standardized processes, data collection and use may not be a silver bullet for improving productivity. Data collection typically works best in standardized environments. So, these differences in how—or whether—firms benefits from data-driven practices make sense to me. Also, if this approach worked equally for all firms, everyone would do it—and no one would gain an advantage. Being data-driven is not a panacea. It is just a potentially powerful tool—one that may work better for certain firms or in certain situations.   

Based on my prior research, I suspect that an important determining factor is how much firms are able to change their processes and culture to accommodate a greater focus on data and analytics. It may come down to whether DDD is a small change the firm or a major process innovation. 

Q. Your earlier research focused on process innovation. How do you define it? Can you provide an example?

Business processes are just the set of activities that a firm engages in to bring a product or service to its customers.  Some of my research has focused on how firms buy and sell goods, and how the Internet radically changed this process for many firms. 

Take the example of purchasing office supplies. Firms lose lots of money when employees make lots of small purchases of paper, pens, folders, etc. on their own. Consolidating the purchases into larger batches and even requiring that formal purchase orders get faxed to pre-approved vendors can be a significant process improvement. But this is not really an innovation. A business process innovation is a major overhaul that can disrupt daily business flows, creating uncertainty and risk for the firm. To return to the example of ordering suppliers, moving the entire process to an online platform that links to a range of digital catalogs, aggregates orders, electronically executes payment, and provides visibility to shipment requires a bigger change. It relies on new information flows, new worker responsibilities, new technology, new skills, and can reveal new failure points. This is what constitutes a true business process innovation. It generates greater risks, but typically also provides greater benefits. 

Q. Business process innovation sounds like a good thing. But your research suggests that not all firms embrace it.

Large firms can be the greatest beneficiaries of these big changes, but it is very difficult to predict ahead of time whether a particular innovation is going to be better for the firm undertaking it, or for its competitors. That’s because attackers in a market come in with a clean slate; they’ve got nothing to lose. They can adopt the ‘latest and greatest’ processes, or customize something new for themselves without having to change existing ways of doing business. Typically, larger firms typically are less nimble and have more to lose. So, there is this real tension and trade-off involved. 

To return to the purchasing example, large firms save the most money from moving their procurement activities online. At the same time, it may be harder for them to replace their existing process flow. 

In fact, I found that the trade-offs came out differently for different types of processes. Purchasing inputs (of more than office supplies) turned out to be readily embraced by large, leading firms. The change involved was not as costly as the benefits it provided. On the other hand, selling final goods over the Internet was another matter entirely. Leaders were far less willing to jump on the e-selling bandwagon.

Q. Why was that? What was different about E-selling?

The difference was that e-buying was not very disruptive for either the adopting firm or its business partners. But e-selling had a significant impact on customers. What I learned in my research was that, any time you start to innovate within a business’s ‘ecosystem’, you have to think about how all the other actors involved may have to change what they do. And, when big firms ask their customers—potentially many—to change their behaviour and/or technology, that can get really hard. If customers aren't ready for the new technology on their own terms, this can prevent connected firms from adopting it, as well. Powerful industrial customers that were not tech-savvy created a hurdle to the uptake of e-selling.

More generally, you really have to take a systems view and keep in mind that certain changes might pose problems for other actors in the value chain. If those firms also have power, this will slow down innovation by other ecosystem players. Quite simply, if the customer is not technologically savvy, it’s going to hold back the entire value chain from using new technology efficiently.

Q. Tell us a bit about your work with the MIT Initiative on the Digital Economy (IDE).

This is a group that got together under the leadership of MIT Professor Erik Brynjolfsson, who is an influential thinker in the realm of IT and digital transformation. There are many challenges coming at us with respect to digital technologies, and we can’t just sit back and let it happen. Those of us affiliated with the IDE feel that we all must embrace the challenges of our time, which includes thinking about how best to educate people and build institutions that are adaptive and robust to the technological transformations we are experiencing We have to ensure we have social policies that make the technology work for us and construct a society that we all want to be a part of—where  people have jobs that are useful and meaningful to them, even as “brilliant machines” are increasingly capable of doing that work for us. 

This group [which also includes Rotman Professor Joshua Gans] is using the tools we have at our disposal to engage with government, social policy makers, and other thought leaders to grapple with these digitally-driven challenges, both at work and in the social sphere. It has been very rewarding to participate in the stream devoted to how IT and data can be a force for good in firms. 

It is not a given that the current ‘data craze’ is a positive development for all concerned. The fact is, there’s always adjustment when new technologies come into play. Academics can sit in their ivory towers and talk about the ‘friction’ involved in adapting to technological change; but on a daily basis, friction means that people are losing their jobs, firms are wasting resources, and these losses can build on themselves to create devastating long-term effects. We’re trying to help to reduce some of this friction by applying creative thinking and evidence-based analysis to what we’re all facing in this new digital age.

 Guiding Principles for Embracing Data-Driven Decision Making

Six guiding principles that run through the experiences of senior leaders who have wrestled with this challenge:

1. Analytics should start with business problems.

2. Analytics requires skilled interpreters.

3. Analytics requires data scientists of different flavours.

4. The analytics team should help get the job done.

5. All leaders need a working knowledge of data science.

6. Analytics should have a seat at the top table.

Kristina McElheran is an Assistant Professor of Strategic Management at the Rotman School of Management and a Digital Fellow at MIT’s Initiative on the Digital Economy.