Big Data is where the paying jobs are. An update posted this week by Forbes Contributor Louis Columbus indicates that the “The [median] advertised salary for technical professionals with Big Data expertise is $104,850 net of bonuses and additional compensation.” And the demand is significant. He points out that
Many of these positions are in IT. That’s great for IT people, right? Not necessarily.
One of the many challenges associated with Big Data is the challenge of making some money from it. That often means taking advantage of data analytics (which comes in many varieties, including data mining, data science, web analytics or even good old statistics). For IT professionals who haven’t learned to cooperate with data analysts, these roles are a real test of skills and cultural adaptability.
In the course of more than a decade guiding organizations beginning or expanding the use of analytics, I have spoken with thousands of analysts and business managers. The majority of these people share a common complaint: they are unsatisfied with the level of support they are able to get from their IT organizations.
They view IT as a hindrance, an obstacle, something to work around. Many of them resort to secretive maneuvers, often violating policy to avoid cooperation, or even contact, with their own IT staff.
These analysts are not doing themselves any favors. Often, they end up working with datasets that are poorly prepared, undocumented and easily lost. Some report spending as much as 80-90% of their time on data preparation.
As analysts grow in numbers and influence, IT professionals must adapt their needs. In order to support data analysts well, you must understand what they need.
Analysts need large volumes of data.
Many roles call for the use of just a bit of data at a time. A customer service staff member may handle just one transaction at any given moment, or perhaps one customer’s transaction history. But analysts are constantly examining groups of transactions, groups of customers. It is common for analysts to work with data for thousands, even millions of individuals in a single analysis.
Analysts need detailed raw data.
Never dispose of data without a good reason. “Nobody will ever need that” is not a good reason. “All they’ll need is a roll-up” is not a good reason. I am routinely forced to tell business people that it is not possible to perform the analysis they desire because they have not saved historic data, or have kept only aggregated data.
Analysts need access to data during their working hours.
Data is the analyst’s raw material. No access to data, no work getting done. Overnight batch jobs do not cut it in the 21st century. Data analysis today is an interactive process.
Analysts need data that is current, complete, consistent and correct.
Data analysts spend most of their time on data prep. They would much prefer to spend most of their time on data analysis.
Analysts need data that is organized appropriately.
In most cases, data for analysts should be organized in flat tables, with rows representing individual people, accounts or transactions. This may not be the best approach for other uses, so you'll need a way to bridge that gap.
Analysts need lots of computing power.
They often don’t know how to estimate their own requirements. This is your opportunity to be a hero, by acting as a go-between with vendor sales or tech support staff to scope those requirements.
Analysts need plenty of storage space.
Nobody wants to waste time or risk destroying something useful while culling files for lack of space.
Analysts need tools that can perform the analysis methods appropriate for the business requirements.
You may not always be able to procure each analyst’s favorite products, but you can always open the process for input and make an effort to understand concerns.
Analysts need help to organize, document and share their work.
Analysts produce valuable intellectual assets every day, but they aren’t very good at managing them. Make yourself a hero by making it easy to protect and exploit those assets.
Yes, there are discussions to be had, details to be worked out. Yet understanding these few points, and accepting them, is enough to make you a stand-out IT professional in the eyes of data analysts and the executives who look to them for business insights.
