Tweet This
I hear the question every day, “Do I need Big Data?” On the contrary, you should use the smallest data you can reasonably manage. So, no, you probably don’t need Big Data. But in some cases, it’s useful.
How Big is it?
A seminal article written by Doug Laney in 2001 described a data problem faced by his clients. They were face with overwhelming quantities (Volume) of data, which were growing quickly (Velocity) and in diverse forms (Variety). His Volume, Velocity and Variety trio caught on, and became widely known as the “3 Vs of Big Data.”
But a lot of people missed Laney’s main point. He was describing problems, not benefits.
Big Data is always challenging and costly to collect, manage and use, and it’s not necessarily relevant to any specific business problem or up to your quality standards. Resources invested in data management are not available for analysis, secondary research, or action, so Big Data may not be a blessing, and may even divert resources away from the data and analysis you need the most.
Think BIG
Some problems are more important than others. Some are more complex. But even a very important, complex problem doesn’t necessarily require a lot of data (by current standards) to solve.
One of my favorite examples is the manned space mission. Think about all the information that must be tracked and all the calculations that are needed to get someone into space and back, alive.
Data needs for a space mission:
- Astronauts’ physical condition and medical information
- Geodesy (spacecraft location) and gravitational fields
- Meteorology – cloud cover and radiation balance
- Atmospheric physics
- Air density from drag and non-gravitation forces
- Ionospheric physics
- Magnetic fields
- Cosmic rays and trapped radiation
- Electromagnetic radiation (UV, X-ray and gamma)
- Interplanetary medium
This list is just a sample of what’s needed, and it’s a lot of information! I’m not even sure what all of it means – I dug up the list from an old NASA document. (Seriously, how do you measure interplanetary medium?)
How much data is required for all of that? How much computing power is necessary? See if you can guess. (No peeking.)
I’ve had the good fortune to speak one to one with Lucy Simon Rakov, one of the programmers from the Mercury Project, the first United States’ manned space program. She told me about the computers used on that project. They were powerful machines that offered a whopping offered a whopping 300 kilobytes of memory. That’s kilobytes. Not exa, not peta, not even mega, but kilo – bytes.
In other words, if you’re very clever, you can run operate a spacecraft on less memory than it takes to store a snapshot of your kid. You can do a lot with a little!
Still, Big Data can sometimes be very valuable, and worth all the fuss and costs required to handle it.
What’s good about Big?
Big Data that provides actionable individual detail can help you make more decisions in quantity. Thousands of decisions, millions of them, in a mere moment.
Imagine that you could observe individuals one by one. You’d know more about their habits, likes and wants. You’d know what to expect from each person. Big Data, if it’s the right data and of good quality, lets you “observe” individuals and treat them as individuals, just as you would in person.
You can use those personal details in the data to provide individualized services, offers and so on. As a consumer, you’ve experienced this through product suggestions like the ones offered on Amazon or Netflix, and perhaps also through other channels like online matchmaking.
But do I need Big Data?
Unless you’re already making very good use of ordinary, small-scale data in your work, you don’t need Big Data right now. Chances are you just need to start using data, period, or make better use of what you have.
