Hacker News Pipeline

  • Guided Project
  • 0.2 hours
  • Intermediate
  • Python

Practice using Python to process JSON data, clean text, and build a data pipeline to summarize Hacker News posts.

Start this project

Overview

In this project, you'll take on the role of a data engineer tasked with processing and summarizing Hacker News posts from 2014. Using Python, you'll read in JSON data, filter the most popular posts, clean the text fields, and build a data pipeline to find the most frequent words in the titles. This hands-on project allows you to apply essential data engineering skills, including parsing JSON, cleaning text data, and constructing an efficient data pipeline. You'll gain experience working with real-world data while building a practical tool to summarize large datasets. Some familiarity with Python is recommended. Objective: Use Python to process Hacker News data and build a data pipeline that identifies the top keywords in post titles from 2014.

What You'll Learn

  • ✓ Process JSON API data in Python.
  • ✓ Expand your portfolio by building a data pipeline

Before You Start

  • ✓ Understanding functional programming basics in Python
  • ✓ Building simple data pipelines using the functional paradigm
  • ✓ Implementing a pipeline API using advanced Python concepts like decorators
  • ✓ Adding dependencies and a scheduler to Python data pipelines

Project Steps

9 steps

  1. 1 Introduction to the Data
  2. 2 Loading the JSON Data
  3. 3 Filtering the Stories
  4. 4 Convert to CSV
  5. 5 Extract Title Column
  6. 6 Clean the Titles
  7. 7 Create the Word Frequency Dictionary
  8. 8 Sort the Top Words
  9. 9 Next Steps
Start this project

Join 1M+ data learners on Dataquest.

  1. 1

    Create a free account

  2. 2

    Choose a learning path

  3. 3

    Complete exercises and projects

  4. 4

    Advance your career

Start learning today

Sign Up For Dataquest Continue Learning