Course

Analyzing Large Datasets in Spark and Map-Reduce

In this course, you’ll learn how to use Apache Spark and the map-reduce technique to clean and analyze large datasets.

Enroll for free

Part of the Data Scientist (Python) path.

4.8 (359 reviews)

8,846 learners enrolled in this course.

Intermediate friendly
3 hours
Self paced
6 lessons

“I really love learning on Dataquest. I looked into a couple of other options and I found that they were much too handhold-y and fill in the blank relative to Dataquest’s method. The projects on Dataquest were key to getting my job. I doubled my income!”

Victoria E. Guzik

Associate Data Scientist @Callisto Media

Analyzing Large Datasets in Spark and Map-Reduce

Enroll For Free

Course overview

Big data is all around us, and Spark is quickly becoming an in-demand Big Data tool that employers want to see.
In this course, you’ll learn the advantages of Apache Spark. You’ll learn concepts such as Resilient Distributed Datasets (RDDs), Spark SQL, Spark DataFrames, and the difference between pandas and Spark DataFrames.
You’ll also learn how to install Spark and PySpark, a Python API that allows you to interact with Spark using Python code. You’ll learn how to integrate PySpark with Jupyter Notebook so you can analyze large datasets.
Best of all, you’ll learn by doing – you’ll practice and get feedback directly in the browser. You’ll work with a variety of real-world datasets, including the text of Hamlet, census data, and guest data from The Daily Show.

Key skills

Breaking down tasks using the map-reduce framework
Processing and transforming larger, raw files using Spark
Working with large, unstructured datasets using Spark SQL and Spark DataFrames

Course outline

Analyzing Large Datasets in Spark and Map-Reduce [6 lessons]

Introduction to Spark 1h

Lesson Objectives

Explain the history of big data
Define how the RDD object works in Spark
Count using Spark

Project: Spark Installation and Jupyter Notebook Integration 1h

Lesson Objectives

Install Spark and PySpark
Integrate PySpark with Jupyter Notebook

Transformations and Actions 1h

Lesson Objectives

Read TSV files into Spark
Apply lambda functions over RDD objects

Challenge: Transforming Hamlet into a Data Set 1h

Lesson Objectives

Transform data from text files into RDD objects
Clean data using lambda functions

Spark DataFrames 1h

Lesson Objectives

Employ Spark DataFrames
Identify the difference between pandas and Spark DataFrames
Perform basic filters with Spark DataFrames

Spark SQL 1h

Lesson Objectives

Query Spark DataFrames using SQL
Employ multiple tables in Spark SQL

The Dataquest guarantee

Dataquest has helped thousands of people start new careers in data. If you put in the work and follow our path, you’ll master data skills and grow your career.

We believe so strongly in our paths that we offer a full satisfaction guarantee. If you complete a career path on Dataquest and aren’t satisfied with your outcome, we’ll give you a refund.

Master skills faster with Dataquest

Go from zero to job-ready

Learn exactly what you need to achieve your goal. Don’t waste time on unrelated lessons.

Build your project portfolio

Build confidence with our in-depth projects, and show off your data skills.

Challenge yourself with exercises

Work with real data from day one with interactive lessons and hands-on exercises.

Showcase your path certification

Share the evidence of your hard work with your network and potential employers.

Grow your career with
Dataquest.

98%

of learners recommend

Dataquest for career advancement

4.85

Dataquest rating

SwitchUp Best Bootcamps

$30k

Average salary boost

for learners who complete a path

Aaron Melton

Business Analyst at Aditi Consulting

“Dataquest starts at the most basic level, so a beginner can understand the concepts. I tried learning to code before, using Codecademy and Coursera. I struggled because I had no background in coding, and I was spending a lot of time Googling. Dataquest helped me actually learn.”

Jessica Ko

Machine Learning Engineer at Twitter

“I liked the interactive environment on Dataquest. The material was clear and well organized. I spent more time practicing then watching videos and it made me want to keep learning.”