Spark DataFrames

At the beginning of this path, you learned that data can be stored inside pandas dataframes, and then you learned data can also be stored inside a SQL database. While these data storage structures are ideal for some data, they're not ideal for massive amounts of data as the data processing workload can get really slow with big datasets using these tools.

In this online Spark dataframes lesson, you'll continue using PySpark and learn about Spark dataframes and their advantages over pandas dataframes. The Spark dataframe was inspired by pandas and combines the scale and speed of Spark with the familiar query, filter, and analysis capabilities of pandas. Spark dataframes also allow us to modify and reuse our existing pandas code to scale it up for handling much larger data sets.

To facilitate your learning about Spark dataframes, you will work with a JSON file containing adata from the 2010 U.S Census. In addition to working with U.S Census data and learning about Spark dataframes, you’ll get to apply what you’ve learned from within your browser so that there's no need to use your own machine to do the exercises. The Python environment inside of this course includes answer checking so you can ensure that you've fully mastered each concept before learning the next concept.


  • Learn how to work with Spark dataframes.
  • Learn the difference between pandas and Spark dataframes.
  • Learn to perform basic filters with Spark dataframes.

Mission Outline

1. The Spark DataFrame: An Introduction
2. Reading in Data
3. Schema
4. Pandas vs Spark DataFrames
5. Row Objects
6. Selecting Columns
7. Filtering Rows
8. Using Column Comparisons as Filters
9. Converting Spark DataFrames to pandas DataFrames
10. Next Steps
11. Takeaways


Course Info:


The median completion time for this course is 6 hours. View Details

This course requires a premium subscription and includes five missions, and one installation tutorial.  It is the 31st course in the Data Scientist In Python path.


Take a Look Inside

(function(d) { d.addEventListener("DOMContentLoaded", function() { var pathname = d.location.pathname.replace(/^[/]|[/]$/g, "").replace("/", "-"); var tags = d.getElementsByTagName("iframe"); var type = pathname.startsWith("course") ? "?course=" : pathname.startsWith("path") ? "?path=" : null; if (type) { var i; for (i = 0; i < tags.length; i++) { if (tags[i].src.indexOf("signup#iframe") !== -1) { tags[i].src = tags[i].src.replace("#iframe", "") + type + pathname + "#iframe"; } } } }, false); })(document);