This feature first appeared in the Spring 2019 issue of Certification Magazine. Click here to get your own print or digital copy.

Humans have always loved large amounts of data. The historian Livy, writing at the dawn of Rome's imperial age (27 B.C. to 476 A.D.) needed 142 volumes for his famous History of Rome Since Its Foundation. At its peak, the widely venerated Library of Alexandria in Ptolemaic Egypt is believed to have housed as many as 400,000 ancient texts.

Even when we aren't attempting to tell the story of a massive civilization over a span of nearly eight centuries, or collect and archive the sum total of human knowledge, humankind likes to have vast caches of data lying around, just in case. The so-called Death Master File (DMF) of the U.S. Social Security Administration currently contains more than 94 million individual records.

Governments, churches, businesses, and other organizations keep detailed records about pretty much everything. Even the random Utah Jazz fan who screamed at Russell Westbrook during an NBA game in Salt Lake City on March 12 had a Twitter account (quickly deleted after his conduct got him a lifetime ban from attending Jazz games) preserving his, er, wisdom for the ages.

Human hands are needed to guide the storage, management, and analysis of Big Data, and more than a few skilled professionals are already on the job. The U.S. Bureau of Labor Statistics estimated last year that there are already 119,500 jobs in the United States for database administrators, or individuals who use specialized software to store and organize data.

Growth in the field over the next 10 years is projected at 11 percent, meaning that an estimated 13,700 more jobs will be created by 2026. And that's just the outlook for people who do the heavy lifting required to pull various types of data together in one place and impose an organizational structure on it. Big Data encompasses many other job roles that are also in high demand.

Data scientist is frequently cited as being among the most sought after, highest compensated job roles in IT, and there's plenty of anecdotal evidence to support that view. Job search site Indeed, for example, as of mid-March, listed 33,776 openings seeking some form of data science expertise, while competitor site Glassdoor had 27,565 listings of its own.

There's opportunity to be had, in other words, along with excellent pay and benefits. Data architects — that's another Big Data job role — build complex data storage systems for companies and organizations. According to salary analytics site Payscale, the average annual earnings figure for a data architect is $113,402, with top-end earners taking home as much as $153,000 per year.

Big Data challenges

Big Data can, of course, be complex and challenging. Sometimes a firm or organization consults its data expecting to find one thing and finds something entirely different. Sometimes there's so much data that it's difficult to know where to begin, or what to look for. For our recently conducted Big Data Certification Survey we decided to investigate some of the ins and outs.

Experts generally characterize Big Data in terms of the so-called five Vs. The five Vs are as follows:

Volume: How much data is there?
Velocity: How quickly is new data being generated?
Variety: What different types of data are in the dataset?
Veracity: How reliable and/or accurate is the data?
Value: What can the data be used to determine or accomplish?

For starters, we asked the more than 250 certified Big Data professionals we surveyed which of the five Vs is most challenging to work with when it comes to turning data into favorable business outcomes. Roughly a third of respondents (32 percent) said that determining Value is the biggest sticking point — it's great that the data is there, in other words, but even some experts still struggle to put it to productive use.

Volume, cited by 24 percent of those surveyed, is also a notable hurdle. Many in IT dislike Big Data as a label, but apparently massive quantity is still enough of an obstacle that we can't just call it data quite yet. Especially in light of the fact that a notable chunk of respondents (20 percent) say velocity, or the speed at which data accumulates, is the biggest challenge to getting what we want from it.

Two things that Big Data professionals may have done a better job of mastering to date are variety and veracity. A notably smaller chunk of survey respondents (just 16 percent) think that working with different types of data is the biggest challenge to productive outcomes, while only a handful (8 percent) are primarily concerned about whether Big Data, generally speaking, is accurate and reliable.

One obvious challenge that encompasses both volume and velocity is the sheer bigness of Big Data. In recent decades, technology has gotten much, much better at both capturing and storing huge amounts of digital information. Technology has also, on the other, greatly increased the availability of data and the speed at which it accumulates.

Are we in danger of falling behind? A strong 76 percent of those surveyed are either very concerned (20 percent) or concerned (56 percent) that there's about to be more data than we can productively store and manage. A further 16 percent are at least somewhat concerned, while just 8 percent feel that we're not about to drown, or at least find ourselves adrift, in an uncontainable sea of data.

Job satisfaction

As touched on already, Big Data professionals have a variety of duties and responsibilities. Some design and build data storage infrastructure, while others are charged with managing and organizing existing data caches. Some specialists are involved in sifting through data to find actionable information, while different individuals process that information and use it to recommend courses of action.

There's quite a bit of work to be done, and only so many hours in the day. Are we pushing the current workforce too hard? A bit more than half of those we surveyed either agree (32 percent) or strongly agree (20 percent) that they are overworked. A little more than one-third (36 percent of respondents) took a neutral position, while the remaining 12 percent disagree that they have too much on their plate. (Nobody strongly disagrees with the notion that Big Data professionals are overworked.)

For most certified networking professionals, the tasks they perform are complex and engaging. A solid 64 percent either agree (40 percent) or strongly agree (24 percent) that their work is challenging, with a further 28 percent taking a neutral position. That leaves just 8 percent who disagree that their work is engaging. (Here again, no strong disagreement was registered.)

We did ask one question that touches on the broad issue of compensation. Generally speaking, are certified Big Data professionals satisfied with their current salary? A bit more than one third either agree (28 percent of respondents) or strongly agree (4 percent) that their current salary is satisfactory, while 36 percent took a neutral view. The remaining one-third either disagree (24 percent) or strongly disagree (8 percent) that their current salary is satisfactory.

Certification = employment

Certification is still somewhat new to the Big Data realm, in the same way that Big Data is relatively new to IT. There are plenty of jobs to be had, it would seem, for those who come by their knowledge of Big Data without certification. Just 26 percent of respondents say they were required to hold one or more Big Data credentials in order to accept their current job.

Even in cases where certification is not required, however, it could be a factor in any hiring decision that gets made. Asked to estimate the impact of certification on being hired at their current job, 33 percent of certified networking professionals said it was either influential (25.9 percent) or very influential (7.4 percent), with an additional 18.5 percent reporting that certification was at least somewhat influential.

It's also true that many choose to get certified with an eye on future employment. Setting aside the popular rationales of gaining skills and increasing compensation, we asked those surveyed to name the two most important benefits of getting a certification.

Three of the top five responses are directly employment-related. The most popular choice is Gain advanced access to technical data. The next two, however, are Gain qualifications for a future job, or Improve or confirm my qualifications for my current job, with Become eligible for positions of greater responsibility with my current employer rating just behind Gain greater confidence in my own skills.

Workplace and education

Nearly every business has to grapple with data-related challenges is 2019. According to our survey audience, however, a sizeable chunk of the Big Data jobs available are focused in three workplace sectors: government (15.4 percent of those surveyed), software (also 15.4 percent), and business services or business consulting (11.5 percent).

Other popular employment sectors include computer or network consulting (7.9 percent of respondents), telecommunications (7.7 percent), manufacturing (7.5 percent), and finance (7 percent).

For teens and young adults who are considering Big Data as a potential career, definitely don't rule out higher education. Among survey respondents, 42.3 percent pursued their formal education far enough to hold a bachelor's degree, and 30.8 percent went one step further and claimed a master's degree.

There's more information to come from our survey. Over the coming months, we'll be posting additional findings online at CertMag.com, where you can also find ongoing dispatches from our 2019 Salary Survey.

NOTE: We had so much fun creating the following illustration for the magazine, that we preserved the combination of photo and graphic for your enjoyment here.