By Brooks Bell
Big Data is “the next frontier for innovation, competition, and productivity,” says McKinsey & Company. But companies and executives rushing into data collection and analysis expecting immediate payoffs are bound to be disappointed. Most companies are years away from being able to effectively profit from data—and not simply for a lack of technology. Instead, at least three entrenched challenges need to be addressed before Big Data can have real impact.
First is the gut-driven approach to strategy that pervades the business world. Leadership by the “highest paid person’s opinion” is a common organizational weakness that should ultimately be remedied by Big Data. But that will happen only when the mindset also shifts so that this person’s decisions are based on real data, not gut assumptions. Simply having more data will not be enough to overturn this mentality and could even make the transition more difficult.
In his book “The Signal and the Noise,” Nate Silver notes that “if there is a mutual distrust between the weather forecaster and the public, the public may not listen when they need to most.” This "crying wolf" problem holds true for CEOs with data, too. If the analysis is wrong—or worse, if the data itself has been improperly collected—decision makers will lose faith in the information and the employees that supply it.
A second challenge is the talent shortage. Currently, not enough people have the necessary skills to make rigorous use of data. It’s been more than 70 years since American national tech policy progenitor Vannevar Bush famously declared “there is a new profession of trail blazers, those who find delight in the task of establishing useful trails through the enormous mass of the common record.” Yet, there remains a shortfall of as many as 190,000 trained data analysts in the United States, according to McKinsey & Company.
A recent survey of IT professionals by SAS and IDG Research found that 57 percent of respondents said they lacked the skills and experience to properly analyze data. And this lack of confidence in analysis is only part of the deficiency. Those working with data must also be skilled in collecting the appropriate metrics with adequate precision.
Instead of creating an elite group of data scientists reporting directly to the executive suite, business leaders must encourage every level of their organizations to produce and cultivate analysts, teaching essential skills, best practices, and rigor along the way. This will increase transparency, encourage demand for data, and help spread essential proficiency.
Stop wrangling and start profiting
Knowing what to do with data is the third challenge. Even with buy-in and proficiency, it’s not exactly clear what most businesses will accomplish by using Big Data. Collecting a huge data set is ultimately meaningless if it can’t inspire action. Indeed, while generating insights is always important, the hallmark of analytics is actionability. Will organizations continue to wade through the noise of historical data or will they begin using it to make practical predictions and forward-looking decisions?
A cell phone company, for example, might be able to collect vast amounts of consumer data, but unless it’s applied in smart ways that actually affect customer experience—say, in retail stores—it only represents theoretical value. A nationwide retail chain can now target direct-mail campaigns with precision, but if sales floor clerks aren’t equipped to capitalize on the outcome, the opportunity will most often be lost. For Big Data to succeed in a big way, the promise of data must be inspiring for every employee.
The lack of comfort with data is central to this disconnect, but the focus of Big Data itself amplifies it. Jorge Luis Borges described the problem in his 1941 short story “The Library of Babel.” The universe of the story consists of endless library shelves of similar-looking books, each containing a different, random assortment of letters and punctuation. Within this library, all thoughts and events are recorded—but any insight is hidden between countless volumes of nonsense. The librarians in Borges’ story muddle through, near madness, unable to make use of their vast resource. This is not that different from what's happening at many companies rushing to Big Data.
Nate Silver alludes to this dilemma when he argues that “distinguishing the signal from the noise requires both scientific knowledge and self-knowledge: the serenity to accept the things we cannot predict, the courage to predict the things we can, and the wisdom to know the difference.” Data, in other words, can’t be used to uncover a truth. Instead, it informs a hypothesis that must then be validated through execution and testing.
Big Data carries a huge promise. But we need to embrace a belief in the importance of data and incorporate it into every stage of planning and at every level of the organization. Mastering small data is not an alternative to utilizing Big Data; it’s a prerequisite. The emphasis needs to shift to the use of more manageable data, more regularly, by more people. Then businesses will be poised to take actions based on data. Only then will the dream of Big Data become real.
Brooks Bell is the founder of Brooks Bell, Inc., which brings scientific disciplines of testing and optimization to the traditionally subjective field of marketing. She speaks regularly on entrepreneurship, analytics, and marketing, and serves as a judge and mentor for Duke University’s entrepreneurship programs.
