What are the different
data roles?
Data science is a big field, and there are plenty of different jobs you can get. But where do you begin?
Let’s take a look at the primary data science roles and what differentiates them from each other.
The Primary Data Science Roles
Learning data science skills can launch a new career, but unfortunately, just knowing data science isn’t enough to determine what kind of work in data you’d like to do. First, you need to know your options so you can determine which career path is right for you. So, in this definitive career guide, we’ll walk you through the primary data science roles, then we’ll help you discover the role that matches your interests with a brief, interactive quiz. Based on our complete userbase of over 1.2 million learners, we’ve determined that data science learners break down largely as follows: 55% data science, 20% data analysis, 15% business analysis, and 10% data engineering. So, let’s briefly cover those four primary roles before drilling down into the details.
A business analyst is often an entry-level position where data professionals begin their careers. The business analyst’s job is to assess business needs and help determine which organizational strategies do and don’t work. Business analysis is a broad field, with many different applications and opportunities. Working as a business analyst requires curiosity, investigation, strategy, project management, and communications. Business analysis is a great career track for people who love working both independently and with teams, and who appreciate some variety in their daily responsibilities.
A data analyst is a slightly more specific job responsibility. These data professionals examine data from an industry or an organization to answer business questions for other people at their organizations. Given the nature of their speciality, data analysts work with many different teams at a company, including marketing, the C-suite, IT, product development, communications, and more. The data analyst’s primary duty is to help other people answer specific business questions using statistical analysis and data visualization.
A data scientist shares some of the responsibilities of a data analyst, but they also do more advanced work building machine learning models to not only answer business questions but also make accurate predictions based on past data. A data scientist has a bit more freedom in how they work than a data analyst — rather than answering specific business questions, data scientists can pursue their own ideas and conduct their own research to find interesting patterns and trends in data that organizational decision-makers can use to improve business strategies.
A data engineer is responsible for an organization’s data infrastructure. This job requires much less statistical analysis but much more software development and programming. Common tasks for data engineers include building data pipelines to get the latest sales, marketing, and revenue data to data analysts and scientists so they can perform analysis to uncover insights. Data engineers also build and maintain the infrastructure an organization uses to store and access past data.
Now that we have a general idea what each of the four primary data science roles does, let’s drill down into some specifics and see these career opportunities in action.
The Primary Data Science Roles
Business Analyst
Data Analyst
Data Scientist
Data Engineer (Python)
The Business Analyst
Business analysts help organizations determine which strategies do and don’t work. Often an entry-level role, The BA is responsible for assessing business needs and making recommendations to improve operations. A business analyst is part investigator and part strategist — with a dash each of presenter and taskmaster. The varied demands of the job are part of what makes it enticing.
Business analysts are often relatively independent, working alone or in small teams as they move around the company solving problems, so it’s a good career for self-driven individuals who like to have the freedom to approach problems their own way.
Required Skills
- Data analysis
- Data exploration
- Data visualization
- Data storytelling
- Data wrangling
Career Prospects
Every day in the life of a business analyst can be different than the last, involving new challenges that put your skills and knowledge to the test. Whether it’s collecting data, presenting to executives, or planning with stakeholders, the day-to-day requirements of the job won’t be boring. If the promise of novelty and challenge sounds enticing, you won’t be disappointed. However, there’s a diverse set of skills to learn, so you’ll want to narrow down your interests and strengths. After all, you need to know what you’re getting into before you commit to pursuing a career in business analysis.
Common daily tasks for business analysts include analyzing and visualizing data using tools such as Microsoft Power BI, preparing reports as PowerPoint slides or PDFs, meeting with key stakeholders, presenting data, and helping others understand business insights. Because different organizations interpret the role differently, your business analyst title may look something like “financial analyst,” “computer systems analyst,” “market research analyst,” “operations research analyst,” “information security analyst,” and more.
Stats and Info
As of April, 2023, the average business analyst in the U.S. earns a salary of $83,382 per year, with an average cash bonus of $3,500 per year (Indeed.com). New business analysts with less than two years of experience earn an average of $79,521, while business analysts with six to nine years of experience earn an average of $88,328, and those with more than ten years of experience earn an average of $93,418. The top five U.S. companies for data analysts in 2023 include Wellington Management (with reported salaries around $126,401), Moody’s Corporation ($118,948), Pacific Gas and Electric ($109,028), McKinsey & Company ($106,752), and UKG ($106,150). The highest-paying cities in the U.S. for data analysts include Houston, TX, New York, NY, and Plano, TX. There are a number of in-demand certifications a business analyst can get to increase their salary. A Certified Anti-Money Laundering Specialist commands a salary 108.6% higher than average, a certified Information Systems Auditor commands a salary 42.23% higher than average, and a Chartered Financial Analyst commands a salary 27.28% higher than average.
According to the most-recent Zippia estimations, there are 272,851 business analysts in the U.S. alone, with 46% identifying as women and 54% identifying as men — the representation of women in business analysis has risen from 31% in 2010 to 46.13%. The average age of a working business analyst is 43.8, and business analysts are 49% more likely to work for public companies than private ones. The most common education level for a business analyst is a bachelor’s degree (71%), followed by a master’s degree (18%), an associate’s degree (7%), a high school diploma (2%), and other degrees (2%), indicating that formal education is not a deciding factor in the employment of business analysts. 51% of business analysts work at companies employing approximately 10,000 people, while 16% work at companies employing between 1,000-10,000 employees — the other 33% work at companies with fewer with 1,000.
The top industries employing business analysts are technology (27%), Fortune 500 (24%), finance (13%), professional services (10%), healthcare (8%), and health care (5%). According to the Bureau of Labor Statistics, the unemployment rate for business analysts has been declining over time, from a high of 6.1% in 2010 to 3.69%. 25% of all business analysts have been at their present job for less than a year, 44% for less than two years, 11% for less than four years, 11% for less than seven years, 5% for less than ten years, and 5% for less than eleven years.
Business analysis is considered an in-demand field. The Bureau of Labor Statistics predicted in 2021 that employment of data analysts will increase by 11% from 2021 to 2031, which the B.L.S. rates as “much faster” than the average for all occupations.o 2031, which the B.L.S. rates as “much faster” than the average for all occupations.
Case Study — Netflix
As of April, 2023, the average business analyst in the U.S. earns a salary of $83,382 per year, with an average cash bonus of $3,500 per year (Indeed.com). New business analysts with less than two years of experience earn an average of $79,521, while business analysts with six to nine years of experience earn an average of $88,328, and those with more than ten years of experience earn an average of $93,418. The top five U.S. companies for data analysts in 2023 include Wellington Management (with reported salaries around $126,401), Moody’s Corporation ($118,948), Pacific Gas and Electric ($109,028), McKinsey & Company ($106,752), and UKG ($106,150). The highest-paying cities in the U.S. for data analysts include Houston, TX, New York, NY, and Plano, TX. There are a number of in-demand certifications a business analyst can get to increase their salary. A Certified Anti-Money Laundering Specialist commands a salary 108.6% higher than average, a certified Information Systems Auditor commands a salary 42.23% higher than average, and a Chartered Financial Analyst commands a salary 27.28% higher than average.
According to the most-recent Zippia estimations, there are 272,851 business analysts in the U.S. alone, with 46% identifying as women and 54% identifying as men — the representation of women in business analysis has risen from 31% in 2010 to 46.13%. The average age of a working business analyst is 43.8, and business analysts are 49% more likely to work for public companies than private ones. The most common education level for a business analyst is a bachelor’s degree (71%), followed by a master’s degree (18%), an associate’s degree (7%), a high school diploma (2%), and other degrees (2%), indicating that formal education is not a deciding factor in the employment of business analysts. 51% of business analysts work at companies employing approximately 10,000 people, while 16% work at companies employing between 1,000-10,000 employees — the other 33% work at companies with fewer with 1,000.
The top industries employing business analysts are technology (27%), Fortune 500 (24%), finance (13%), professional services (10%), healthcare (8%), and health care (5%). According to the Bureau of Labor Statistics, the unemployment rate for business analysts has been declining over time, from a high of 6.1% in 2010 to 3.69%. 25% of all business analysts have been at their present job for less than a year, 44% for less than two years, 11% for less than four years, 11% for less than seven years, 5% for less than ten years, and 5% for less than eleven years.
Business analysis is considered an in-demand field. The Bureau of Labor Statistics predicted in 2021 that employment of data analysts will increase by 11% from 2021 to 2031, which the B.L.S. rates as “much faster” than the average for all occupations.
Sample Job Posting
Most data analyst job postings are designed to tell you a little about the employer, including their values and philosophies, the job responsibilities, required qualifications, and preferred qualifications. It’s important to familiarize yourself with this format because each employer may interpret “data analyst” differently, so you want to make sure that the company’s vision for the job and your vision for the job align. Below is a sample job posting demonstrating what you’ll typically see — including which details to look for in all of the descriptive language.
Company Overview
Fieldman Consulting is an IT consulting and staffing company bringing experienced experts and professionals together to provide innovative IT solutions tailored to our clients specifications.
Position Description
The business analyst at Fieldman plays a critical part in ensuring that we’re meeting client objectives in several important areas. Primary functions will include the following:
- Gathering and analyzing different types of data, based on client needs and project goals
- Recommending improvements for organizational performance
- Creating effective, easy-to-understand data reports for multiple audiences
- Contributing data and operational analysis to organizational strategy
- Identifying trends and determining critical measurements
- Determining production, quality, and customer-service strategies
- Identifying revenue opportunities and projecting expansion
- Analyzing organizational operations for areas for improvement
- Additional duties as necessary
Qualifications
- Proficiency with Microsoft Excel
- Strong knowledge of SQL
- Familiarity with writing complex queries
- Ability to gather and document requirements and confirm observations with clients
- Ability to perform fit/gap analysis based on requirements
- Ability to create detailed and comprehensive design documents
- Familiarity with testing designs and plans at scale
- Experience researching, analyzing, and validating business requirements
Often, job postings will also include a “preferred qualifications” section, which will follow the required qualifications. These preferences will be unique to each posting and each company. Don’t disqualify yourself for a position simply because you don’t match all of the preferred qualifications. Instead, you can use this section to get a glimpse of what the future in this role might look like for you, as it indicates the organization’s ideal version of this role.
The Data Analyst
A data analyst’s primary job is to look at company or industry data and use it to answer business questions — then communicate those answers to other teams in the company for action. For example, a data analyst might be asked to look at sales data from a recent marketing campaign to assess its effectiveness and identify strengths and weaknesses. This would involve accessing the data, cleaning it, performing some statistical analysis to answer relevant business questions, and then visualizing and communicating the results.
Over time, data analysts often work with a variety of different teams within a company; you may work on marketing analytics one month, then help the CEO use data to find reasons the company has grown the next. Essentially, the data analyst is a generalist who can fit into many roles and teams to help others make better data-driven decisions. You will typically be given business questions to answer rather than asked to find interesting trends on your own, as data scientists often are, and you’ll generally be tasked with mining insights from data rather than predicting future results with machine learning.
Required Skills
- Programming fundamentals
- Database analysis
- Data cleaning
- Data visualization
- Inferential statistics
- Descriptive statistics
- Data storytelling
- Version control
Career Prospects
“Data analyst” is a broad term that encompasses a wide variety of positions, so your career path is fairly open-ended. Data analyst job postings often use different terms for the same position, so you may have a title like business analyst, business intelligence analyst, operations analyst, or database analyst. Regardless of title, the data analyst is a generalist who can fit into many roles and teams to help others make better data-driven decisions. Common daily tasks include data cleaning, performing analysis, and creating data visualizations.
One common next career step for data analysts is to continue building your data science skills — often with a focus on machine learning — and work toward a slightly more advanced role as a data scientist. Alternatively, if you’re more interested in software development, data infrastructure, and helping build a complete data pipeline, you could work toward a position as a data engineer. Some data analysts also use their programming skills to transition into more general developer roles.
If you stick with data analysis, many companies hire senior data analysts. At larger companies with data teams, you can also think about working toward management roles if you’re interested in developing management skills.
Stats and Info
As of April, 2023, the average data analyst in the U.S. earns a salary of $74,433 per year, with an average cash bonus of $2,000 per year (Indeed.com). New data analysts with less than a year of experience earn an average of $70,547, while data analysts with ten years or more of experience earn an average of $88,110. The top five U.S. companies for data analysts in 2023 include Intuit (with reported salaries around $160,000), Meta ($141,000), Capitol One ($102,000), Cisco Systems ($94,000), and AT&T ($93,000). The highest-paying cities in the U.S. for data analysts include Charlotte, N.C., Washington, D.C., and New York City. The most in-demand disciplines for data analysts are probability and statistics (knowledge of which commands a 16.19% higher salary, on average), and 59% of data analysts in the U.S. report salary satisfaction based on the cost of living in their areas.
According to the most-recent Zippia estimations, there are 93,471 data analysts in the U.S. alone, with 49.8% identifying as women and 50.2% identifying as men — the representation of women in data analysis has risen from 46.77% in 2010 to 49.8%. The average age of a working data analyst is 43, and data analysts are 52% more likely to work for public companies than private ones. The most common education level for a data analyst is a bachelor’s degree (65%), followed by a master’s degree (15%), an associate’s degree (12%), other degrees or certifications (5%), and a high school diploma (3%), indicating that formal education is not a deciding factor in the employment of data analysts. 55% of data analysts work at companies employing approximately 10,000 people, while 24% work at companies employing between 1,000-10,000 employees — the other 21% work at companies with fewer with 1,000.
The top industries employing data analysts are Fortune 500 companies (24%), technology (17%), finance (13%), healthcare (8%), and professional services (7%). According to the Bureau of Labor Statistics, the unemployment rate for data analysts has been declining over time, from a high of 6.9% in 2010 to 4.3%. 31% of all data scientists have been at their present job for less than a year, 34% for less than two years, 25% for less than seven years, and 10% for less than ten years.
Data analysis is considered an in-demand field. The Bureau of Labor Statistics predicted in 2021 that employment of data analysts will increase by 23% from 2021 to 2031, which the B.L.S. rates as “much faster” than the average for all occupations.
tions.o 2031, which the B.L.S. rates as “much faster” than the average for all occupations.
Case Study — FedEx
Worldwide shipping giant FedEx is no stranger to data analysis. The company has famously touted the value of data since the late-’70s, when chairman Fred Smith said that information about a package is just as important as the package itself. Fast-forward a few decades, and this respect for data has blossomed into FedEx Dataworks, an internal FedEx agency that analyzes the organizations mountains of data. By analyzing their data, answering business questions, and extracting insights, Dataworks helps FedEx serve its customers better and faster — which is no small feat, given the state and complexity of modern supply chain management.
By drawing data-based conclusions about consumer demand, optimal delivery routes, and the status of deliveries in process, Dataworks offers improvements not only to FedEx’s internal supply chain management but also to its affiliates and partners. That’s a win for everybody, and it’s a big leg up in the competitive world of shipping and fulfillment. Knowing when consumer trends are gaining traction, and when they could deliver on those demands, retailers can plan promotions and know when to tell their customers to buy, buy, buy! Route information, delivery times, and other pieces of fulfillment information can help identify potential locations for retail expansion.
At Dataworks, they believe that every dollar invested into better insights is a dollar gained in sales. With digital commerce increasingly on the rise following the global COVID-19 pandemic and its associated supply chain challenges, shopping is not inextricably tied to shipping. Dataworks uses AI and machine learning to predict delivery dates on the fly, especially as shipping conditions change. This is how today’s online shoppers know how long it will take to get their goods, and it helps determine if they order online (business for FedEx) or in person.
Sample Job Posting
Most data analyst job postings are designed to tell you a little about the employer, including their values and philosophies, the job responsibilities, required qualifications, and preferred qualifications. It’s important to familiarize yourself with this format because each employer may interpret “data analyst” differently, so you want to make sure that the company’s vision for the job and your vision for the job align. Below is a sample job posting demonstrating what you’ll typically see — including which details to look for in all of the descriptive language.
Company Overview
Innovation Inc. is a multichannel performance marketing company that focuses on generating high-value leads in the personal finance field, for topics such as debt relief, financial aid, and insurance. Innovation Inc. believes in helping consumers reach their full financial potential and connecting them with partner services that can help them achieve their personal finance goals.
Position Description
The data analyst manages the analytics solution life cycle to identify, analyze, and present insights that can drive strategic business decisions to create a competitive advantage for Innovation Inc. In this capacity, you will work to anticipate opportunities, manage data analytic platforms, implementing business intelligence solutions, and apply quantitative modeling. Working cross-functionally with members of other teams, you will identify, develop, and recommend actionable insights to support business objectives.
Primary Functions
- Gathering and analyzing various types of data, based on project goals
- Providing recommendations to improve organizational performance
- Assembling and summarizing data to create effective reports
- Supporting organizational goals by contributing analysis to strategic thinking
- Evaluating trends; establishing critical measurements; and helping determine production, quality, and customer-service strategies
- Studying trends and revenue opportunities; projecting expansion prospects; analyzing organization operations; and identifying areas for improvement, cost reduction, and enhanced proficiency
- Additional duties as necessary
Qualifications
- Proficiency with Microsoft Excel
- Subject-matter expertise in analytics tools (Alteryx, SQL), data visualization tools (Tableau), Cloud Platforms (GCP, AWS, Azure), and scripting languages (Python, R)
- Data collection, data mapping, and relational database experience
- Ability to analyze large datasets
- Strong verbal and written communication skills
- Analysis and problem-solving skills
- Attention to detail
- Ability to solve complex problems in a fast-paced environment
- Experience developing and giving presentations
Often, job postings will also include a “preferred qualifications” section, which will follow the required qualifications. These preferences will be unique to each posting and each company. Don’t disqualify yourself for a position simply because you don’t match all of the preferred qualifications. Instead, you can use this section to get a glimpse of what the future in this role might look like for you, as it indicates the organization’s ideal version of this role.
The Data Scientist
Data scientists do many of the same things as data analysts, but they also typically build machine learning models to make accurate predictions about the future based on past data. A data scientist often has more freedom to pursue their own ideas and experiment to find interesting patterns and trends in the data that management may not have thought about.
As a data scientist, you might be asked to assess how a change in marketing strategy could affect your company’s bottom line. This would entail a lot of data analysis work (acquiring, cleaning, and visualizing data), but it would also probably require building and training a machine learning model that can make reliable future predictions based on past data.
Required Skills
- Unsupervised machine learning
- Supervised machine learning
- Machine learning model optimization
- Inferential statistics
- Database statistics
- Advanced programming in Python and R
Career Prospects
A data scientist, like a data analyst, needs to be able to clean, analyze, and visualize data. However, a data scientist will have more depth and expertise in these skills, and will also be able to train and optimize machine learning models. The data scientist will uncover hidden insights by leveraging both supervised (e.g. classification, regression) and unsupervised learning (e.g., clustering, neural networks, anomaly detection) methods toward their machine learning models. They are essentially training mathematical models that will allow them to better identify patterns and derive accurate predictions.
Common daily tasks for data scientists include the following:
- Using machine learning to build better predictive algorithms
- Evaluating statistical models to determine the validity of analyses
- Testing and continuously improving the accuracy of machine learning models
- Building data visualizations to summarize the conclusion of an advanced analysis
After working for a while as a data scientist, your next job title may well be senior data scientist, a position that’ll earn you about $20,000 more per year on average. You might also choose to specialize further in machine learning as a machine learning engineer, which would also bring a pay raise. Or, you can look more toward management with roles like lead data scientist. If you want to maximize earnings, your ultimate goal might be a C-suite role in data — such as chief data officer — although these roles require management skills and may not involve a lot of actual day-to-day work with data.
Stats and Info
As of April, 2023, the average data scientist in the U.S. earns a salary of $130,556 per year (Indeed.com). New data scientists with less than a year of experience earn an average of $110,411, while data scientist with three to five years of experience earn an average of $150,443. The top five U.S. companies for data scientists in 2023 include Stitch Fix (with high-end reported salaries around $228,683), Airbnb ($193,608), Lyft ($184,248), Comcentric ($183,124), and Apple (@173,534). The highest-paying cities in the U.S. for data scientists include Palo Alto, CA, San Francisco, CA, and Boston, MA. The most in-demand skill for data scientists is cloud architecture (knowledge of which commands a 27.28% higher salary, on average), and 68% of data scientists in the U.S. report salary satisfaction based on the cost of living in their areas.
According to the most-recent Zippia estimations, there are approximately 106,973 job openings for data scientists in the U.S. job market alone. Among working data scientists, 20.4% identify as women, and 79.6% identify as men. The most common average age of a working data scientist is 40+ (49%), with those between age 30-40 making up 30% of total working data scientists, and those between age 20-30 comprising 22% of working data scientists. The majority of data scientists work for public companies (53%), with a minority of 44% working for a private company, and 3% working in education. The most common education level for a data analyst is a bachelor’s degree (51%), followed by a master’s degree (34%), a doctoral degree (13%), other degrees or certifications (1%), and an associate’s degree (1%), indicating that formal education is more common among data scientists than data analysts — however, it isn’t necessarily critical for getting a job. 41% of data analysts work at companies employing approximately 10,000 people, while 30% work at companies employing between 1,000-10,000 employees — the other 29% work at companies with fewer with 1,000.
The top industries employing data scientists are technology (19%), Fortune 500 (19%), start-ups (10%), finance (9%), and internet-based businesses (7%). 25% of all data scientists have been at their present job for less than a year, 23% for less than two years, 13% for less than four years, 31% for less than seven years, 5% for less than ten years, and 4% for less than eleven years.
Data science is considered an in-demand field. The Bureau of Labor Statistics predicted in 2021 that employment of data scientists will increase by 36% from 2021 to 2031, which the B.L.S. rates as “much faster” than the average for all occupations.
Case Study — Google
Data science has immediate applications in several industries, including fraud and risk detection, speech recognition, airline route planning, and even gaming and augmented reality. One industry undergoing significant development thanks to data science is healthcare. Fields like genetic analysis and drug development are already reaping the benefits of applied data science, but particularly interesting use case is in medical imaging. And to see this in action, we can look no further than Google.
With the help of new machine learning technologies, like Google’s LYNA (Lymph Node Assistant), doctors can conduct microscopic analysis of cancerous tumors to make critical treatment decisions. A key step in saving lives when it comes to cancer treatment is finding cancer cells that have metastasized from a primary site into the lymph nodes. We know that about 25% of metastatic lymph node staging reviews don’t hold up to a second look. Further, given restrictions on time and available equipment, doctors commonly only detect bout 38% of small metastases.
Data science, specifically LYNA, has changed the numbers in this game. Applied to pathology slides and a dataset offered up by the Naval Medical Center in San Diego, CA, LYNA could distinguish cancerous from non-cancerous slides 99% of the time. Further, it could identify areas of concern that had previously been impossible for doctors to detect. In practice, LYNA created faster, more accurate diagnoses and enabled doctors to create treatment plans with smaller margins of error and great patient success rates.
But LYNA didn’t build itself, and that’s where the critical work of data scientists comes into play. Data science is changing lives around the world every day.
Sample Job Posting
Most data science job postings communicate information about the employer, such as their values and philosophies, the job responsibilities, required qualifications, and preferred qualifications. It’s important to understand this format because each employer may interpret “data scientist” differently, and each section of the posting tells you more about the job than is in the posting itself. Below is a sample job posting demonstrating what you’ll typically see in an entry-level data scientist job posting — including which details to look for in all of the descriptive language.
Company Overview
The Data Market is a managed data science marketplace for big data and analytics consulting. Our consumers are organizations who want to harness the power of their data but lack the resources to do so internally. We match organizations with our expertly curated, third-party consultants to deliver technology and business solutions on demand.
Position Description
We are looking for a skilled data scientist to become a member of our mobile app development team. The successful candidate will develop and implement data-driven solutions to optimize the mobile app’s performance and UX. You will collaborate with product managers, engineers, and designers to isolate areas for improvement and optimization.
Primary Functions
- Analyzing user behavior and data to isolate areas for improvement and optimization
- Developing data models and algorithms to enhance app performance and UX
- Working with other functional teams to generate data-driven solutions
- Deploying data-driven A/B tests to measure new features
- Interpreting data to find insights for other cross-functional teams
- Creating dashboards and reports to manage KPIs
- Identifying emerging trends in mobile app development
Qualifications
- Strong knowledge of data science and statistics
- Experience working with data modeling, machine learning, and statistical analysis
- Knowledge of Python, R, SQL
- Proficiency with Tableau or PowerBI for data visualization
- Knowledge of development frameworks like React Native or Flutter
- Strong written and verbal communication skills
- Adaptable problem-solving skills and the ability to manage multiple challenges at once
Often, you will also see a “preferred qualifications” section detailing attributes above and beyond the baseline qualifications. These preferences are unique to each job posting. If you don’t match all of the preferred qualifications, don’t discard the posting. Instead, treat the preferred qualification as a snapshot of how this role might develop for you, as it indicates the organization’s ideal version of this role.
The Data Engineer
A data engineer manages a company’s data infrastructure. Their job requires a lot less statistical analysis and a lot more software development and programming skill. At a company with a data team, the data engineer might be responsible for building data pipelines to get the latest sales, marketing, and revenue data to data analysts and scientists quickly and in a usable format. They’re also likely responsible for building and maintaining the infrastructure needed to store and quickly access past data.
The data engineer guarantees that data scientists and data analysts have the tools they need to do their jobs. Data engineers are responsible for constructing data pipelines and often have to use complex tools and techniques to handle data at scale. A good data engineer allows a data scientist or analyst to focus on solving analytical problems, rather than having to move data from source to source.
Required Skills
- Data architecture
- Data pipelines
- Data wrangling
- Database analysis
- Big data
Career Prospects
Every company depends on its data to be accurate and accessible to the individuals who need to work with it. The data engineer ensures that any data is properly received, transformed, stored, and made accessible to other users. Commonly daily data engineering tasks include the following:
- Integrating external or new datasets into existing data pipelines
- Building APIs for data consumption
- Applying feature transformations for machine learning models on new data
- Continuously monitoring and testing the system to ensure optimized performance
Data engineers can move into more senior engineering positions through continued experience, or they can use their skills to transition into a variety of other software development specialties. Outside of specialization, there is also the potential to move into management roles, either as the leader of an engineering or data team (or both, although only very large companies are likely to have a sizable data engineering team).
Stats and Info
As of April, 2023, the average data engineer in the U.S. earns a salary of $133,279 per year (Indeed.com). However, over time, a data engineer’s salary increases significantly — the average salary for a data engineer with ten years of experience is $171,754. The top U.S. company for data engineer salaries is Fidelity Talent Source, where the average of reported salaries is $245,557 per year. The highest-paying cities in the U.S. for data analysts include Washington, D.C. ($167,729), San Francisco, CA ($153,527), and McLean, VA ($144,515). The most in-demand skill for data engineers is Redis (knowledge of which commands a 15.34% higher salary, on average), and the top certification is Certified Authorization Professional (CAP), which brings a 41.93% salary increase, on average. 69% of data analysts in the U.S. report salary satisfaction based on the cost of living in their areas.
According to the most-recent Zippia estimations, there are currently 303,105 available data engineering jobs in the U.S. alone. Among working data engineers 19% identify as women and 82% identify as men. The average age of a working data engineer is 39, and data engineers are 56% more likely to work for public companies than private ones. The most common education level for a data analyst is a bachelor’s degree (65%), followed by a master’s degree (22%), an associate’s degree (7%), a doctoral degree (2%), and other degrees or certifications (4%), indicating that formal education is not the only deciding factor in the employment of data engineers. 40% of data analysts work at companies employing approximately 10,000 people, while 31% work at companies employing between 1,000-10,000 employees — the other 29% work at companies with fewer with 1,000 employees.
The top industries employing data engineers are technology (24%), Fortune 500 (18%), finance (10%), internet services (10%), and professional services (7%). According to the Bureau of Labor Statistics, the unemployment rate for data engineers plummeted over time, from a high of 4.6% in 2010 to 2.1%. 23% of all data engineers have been at their present job for less than a year, 32% for less than two years, 14% for less than four years, 21% for less than seven years, and 5% for less than ten years.
Data engineering is considered an in-demand field. Job growth in this sector is predicted to increase by 21% between 2018 and 2028, and data engineer salaries have increased 10% in the last ten years.
Case Study — Young Alfred
Young Alfred may be a newish player in the online insurance marketplace game, but they established themselves quickly. They developed an excellent reputation as a trustworthy resource for finding the best home insurance policy, and they continue to gain popularity. Insuranks has them at a 4.6 out of 5 for customer ranking and an A+ rating with the Better Business Bureau. It’s safe to say that plenty of people have come to rely on this service for their insurance needs.
Naturally, Young Alfred collects a lot of data about its consumers to be able to effectively match them with the best policies. But it isn’t enough just to analyze that data — Young Alfred needs to release new, attractive tools to remain the most relevant resource for their consumers. A good example is the home insurance calculator, which uses minimal input data to estimate a home insurance premium — the less users have to provide, and the faster than can get results, the more attractive the tool. However, a comprehensive home insurance calculation involves over 100 factors — a number most users would balk at. Take into account that there are 42,000 zip codes in the U.S. and each one needs a custom calculator to function properly. Now, you’ve got a challenge.
Instead of using a clustering algorithm, the data engineers at Young Alfred solved this problem by choosing a simpler model as a baseline and improving it over time. Essentially, they built a simple model framework with 42,000 sets of model coefficients. Ultimately, the engineers finalized a model that cannot guarantee that it will get all 10 of its input factors and still perform while minimizing squared error. Basically, after evaluating dozens of model types to solve their problem, the team implemented a model that they knew was simple at its core and would allow them to iterate and improve as necessary to keep up with their growing calculator needs. And while it’s not perfect, it’s one of the most powerful insurance-policy-matching machines out there, thanks to its dedicated team of data engineers.
Sample Job Posting
Most data engineer job postings will be packed with information in a small space, including details about the employer, their values and philosophies, the job responsibilities, required qualifications, and preferred qualifications. It’s important to become familiar with this format because each employer may interpret “data engineer” differently, so you want to make sure that you can extract important details about the job from any posting. Below, we’ve provided a sample data engineer job posting so you can see where the important details are laid out.
Company Overview
Aquarian Securities is a global investment firm delivering financial strategy for organizations, financial professionals, and investors worldwide. With $24.5 billion AUM and $1.2 billion AUA as of 31 April 2023, we offer mutual funds, closed-end funds, institutional accounts, and separate accounts for high-net-worth investors.
Position Description
The Data Engineer at Aquarian Securities will work with an active and collaborative team in the IT Department designing, creating, and optimizing strategies for data collection, storage, and access.
Primary Functions
- Design, build, test, and maintain both on-premises and cloud-based data pipelines
- Develop data strategies and improve data products and quality
- Work with analysts to explore datasets and contribute to business objectives
- Support data analysts, including statistical and predictive analytics, as well as machine learning models
- Research new technologies and opportunities to optimize data acquisition and enablement
- Uphold industry best practices for system security, privacy, classification, performance, and retention
Qualifications
- Industry experience with asset management
- Experience designing, building, and optimizing solutions with Microsoft SQL Server technologies
- Knowledge of how to create data pipelines and ETL/ELT processes
- Ability to understand and explain complex data models
- Basic administrative tasks like backup and recovery, index maintenance, etc.
- Familiarity with Microsoft Power BI and SQL Server Reporting Services
- Detail-oriented with excellent written and verbal communication skills
- Occasional availability outside standard business hours
Often, job postings will also include a “preferred qualifications” section, which follows the required qualifications. These preferences will be unique to each posting and each company. Don’t think that the preferred qualifications are a way to discourage you if you don’t meet them all. Instead, you can use this section to get a glimpse of what the future in this role might look like for you, as it indicates the organization’s ideal version of this role.
Achieve your career
goals in 2023
Take control of your career with a premium subscription today. Access our full catalog, and complete even more projects to make your portfolio stand out from the crowd.