Course

You'll learn how to:

Start this course today

Build hands-on data skills with interactive exercises and projects.

Sign up

About this course

In our Processing Large Datasets in Pandas course, you’ll learn how to work with medium-sized datasets in Python by optimizing your pandas workflow, processing data in batches, and augmenting pandas with SQLite.

In this course, you’ll learn to reduce the memory footprint of a pandas dataframe while working with data from the Museum of Modern Art. You’ll also learn how to work with dataframe chunks and how to use them to increase processing speed in pandas. You will also get the chance to practice working with dataframe chunks and optimize dataframe types while exploring data from the Lending Club.

After learning about optimizing dataframes and working with dataframe chunks, you will learn how to augment pandas with SQLite to combine the best of both tools. We’ll cover when to use disk space over in-memory space, as well as how to run SQL queries using pandas.

At the end of the course, you’ll complete a project in which you will work on a real-life example of using the pandas SQLite workflow to analyze startup fundraising deals using data from CrunchBase. This project is a chance for you to combine the skills you learned in this course and analyze startup fundraising deals using a new workflow. It will also serve as a portfolio project that you can showcase to your future employer so they can feel confident in your data engineering and SQLite skills!

By the end of this course, you’ll be able to:

  • Learn how to reduce the memory footprint of a pandas DataFrame.
  • Explore how to process large DataFrame in chunks and using SQLite.

Lessons in this course

Loading lessons....

Thousands of learners have changed their careers with Dataquest

97%

Learners who recommend
Dataquest for career advancement

4.9 stars

Dataquest rating on
G2Crowd and SwitchUp

$30k

Average salary boost
for learners who complete a path

Join a community of 1M+ data learners on Dataquest

1

Sign up for a free account

Get access to hundreds of free lessons.

2

Choose a course or path

Start anywhere, from beginner topics to advanced concepts.

3

Learn with hands-on exercises

Learn with real data and build your experience.

Apply your skills

Create projects, build your portfolio, and build your career.

Sign up today

or