Course overview
Big data is all around us, and Spark is quickly becoming an in-demand Big Data tool that employers want to see.
In this course, you’ll learn the advantages of Apache Spark. You’ll learn concepts such as Resilient Distributed Datasets (RDDs), Spark SQL, Spark DataFrames, and the difference between pandas and Spark DataFrames.
You’ll also learn how to install Spark and PySpark, a Python API that allows you to interact with Spark using Python code. You’ll learn how to integrate PySpark with Jupyter Notebook so you can analyze large datasets.
Best of all, you’ll learn by doing — you’ll practice and get feedback directly in the browser. You’ll work with a variety of real-world datasets, including the text of Hamlet, census data, and guest data from The Daily Show.
Key skills
- Breaking down tasks using the map-reduce framework
- Processing and transforming larger, raw files using Spark
- Working with large, unstructured datasets using Spark SQL and Spark DataFrames
Course outline
Analyzing Large Datasets in Spark and Map-Reduce [6 lessons]
Introduction to Spark 1h
Lesson Objectives- Explain the history of big data
- Define how the RDD object works in Spark
- Count using Spark
Project: Spark Installation and Jupyter Notebook Integration 1h
Lesson Objectives- Install Spark and PySpark
- Integrate PySpark with Jupyter Notebook
Transformations and Actions 1h
Lesson Objectives- Read TSV files into Spark
- Apply lambda functions over RDD objects
Challenge: Transforming Hamlet into a Data Set 1h
Lesson Objectives- Transform data from text files into RDD objects
- Clean data using lambda functions
Spark DataFrames 1h
Lesson Objectives- Employ Spark DataFrames
- Identify the difference between pandas and Spark DataFrames
- Perform basic filters with Spark DataFrames
Projects in this course
Project: Spark Installation and Jupyter Notebook Integration
Learn how to set up PySpark and integrate it with Jupyter Notebook.
The Dataquest guarantee
Dataquest has helped thousands of people start new careers in data. If you put in the work and follow our path, you’ll master data skills and grow your career.
We believe so strongly in our paths that we offer a full satisfaction guarantee. If you complete a career path on Dataquest and aren’t satisfied with your outcome, we’ll give you a refund.
Master skills faster with Dataquest
Go from zero to job-ready
Learn exactly what you need to achieve your goal. Don’t waste time on unrelated lessons.
Build your project portfolio
Build confidence with our in-depth projects, and show off your data skills.
Challenge yourself with exercises
Work with real data from day one with interactive lessons and hands-on exercises.
Showcase your path certification
Impress employers by completing a capstone project and certifying it with an expert review.
Learning resources
AI is reshaping the tech landscape, so what does that m...
ChatGPT is an artificial intelligence chatbot developed...
With the ever-growing applications of Artificial Intell...
Introduction Deep learning is a rapidly growing subfiel...
Grow your career with
Dataquest.
97%
of learners recommend
Dataquest for career advancement
4.9
Dataquest rating on
G2Crowd and SwitchUp
$30k
Average salary boost
for learners who complete a path
Join 1M+ data learners on
Dataquest.
1
Sign up for a free account
2
Choose a course or path
3
Learn with hands-on exercises
4