Big Data is:
- Immediate - in the sense that you need to do something about it now
- Intimidating - what if you don’t?
- Ill-defined - what is it, anyway?
This is what Vance Loiselle, CEO of analytics company Sumo Logic recently told me. With a nod to the well-known 3 V's of Big Data (volume, velocity, and variability), I have coined these the 3 I's of Big Data.
The definition of Big Data may still be up for debate. But with overall corporate data nearly doubling year over year, the number of Facebook users exceeding 900M, and Twitter tweets blowing through 400M per day, two things about Big Data are certain. As Loiselle put it, "Big Data is not going away and it’s only going to get bigger."
So let's explore the 3 I's of Big Data. As always, I welcome your comments here and at dave@vcdave.com.
1. Ill-defined: What is Big Data?
Gartner analyst Doug Laney has characterized Big Data as "data that’s an order of magnitude greater than data you’re accustomed to."
Ed Dumbill, program chair for the O'Reilly Strata Conference, describes Big Data as, "data that exceeds the processing capacity of conventional database systems. The data is too big, moves too fast, or doesn't fit the strictures of your database architectures. To gain value from this data, you must choose an alternative way to process it."
Another way to view Big Data is that it's a transformative set of technological advances that have made analyzing data vastly more efficient.
Consumer facing companies like Google and Facebook have driven many of the recent advances in Big Data efficiency. Facebook has some 900 million users and is still growing, while some estimates put the number of search queries Google handles at 3 billion per day. Twitter handles some 400 million tweets per day.
In an ironic twist, highlighted by cloud cost management vendor Cloudyn, increased efficiency doesn't drive down usage. It increases it.
Known as Jevons Paradox, it's named for the economist who made the observation about the Industrial Revolution. Similarly, as technological advances make storing and analyzing data more efficient, companies are doing a lot more analysis -- not less. This, in a nutshell, is Big Data.
2. Intimidating: How do you make Big Data approachable?
There are lots of challenges in leveraging Big Data, from managing the data to having the right tools to get you the insights that matter.
Fortunately, Big Data Apps are springing up all over the place to make Big Data a lot easier to take advantage of.
Companies like Splunk and Sumo Logic are Big Data Apps for machine data. Marketing relevance company BloomReach is another such example. The company processes more than 100 million web pages, generating 94% average annual incremental traffic as a result.
3. Immediate: What's actionable about big data?
Technological improvements that increased the efficiency of coal use led to increased consumption of coal in a wide range of industries, fueling the Industrial Revolution. In much the same way, technological advances that are increasing the efficiency of analyzing and storing data are driving a Big Data Revolution:
- A lot more data is being generated. While humans generate a seemingly large amount of data in the form of photos and emails, that data production is limited by the number of people. That amount of data is dwarfed by "sensor" data generated by machines--data from computers and network devices, from airplanes, from cell phones, and from connected GPS devices, for example. And high bandwidth wireless networks are now in place to transport that data back to data centers for storage and analysis.
- Technologies created by companies serving an unprecedented number of consumers have driven efficiencies in how data is stored and analyzed. You now have the ability to store and analyze vastly more data than you could in the past.
- You can setup your own computer resources to store and analyze data, but the availability of scaleable cloud computing resources like Amazon Web Services means you can access the resources necessary to do large scale data analysis quickly and easily.
The next step in making big data actionable is to make Big Data truly immediate by reducing the time between when data is collected and when you get insights from that data. As J. Andrew Rogers, founder and CTO of Space Curve put it, "the analytic value of data decays rapidly." That means being able to analyze your data as fast as possible is critical to gaining competitive advantage.
David Feinleib is the author of the Big Data Landscape. He provides research and advisory services. For more information, contact him at dave@vcdave.com.
