Distributed Data Processing with PySpark

Distributed Data Processing with PySpark [2 courses]

Learn distributed data processing with Apache Spark and PySpark.

• Course 1

  Analyzing Large Datasets in Spark

  3h

  Work with Apache Spark to process massive datasets using RDDs, DataFrames, and Spark SQL across distributed environments.

  Course Objectives ▾

  • Set up and configure Spark applications using SparkSession
  • Transform and analyze distributed datasets using RDDs and DataFrames
  • Write SQL queries on large datasets using Spark SQL
  • Understand Spark's architecture including Drivers, Executors, and lazy evaluation
• Course 2

  PySpark for Data Engineering

  6h

  Move beyond notebooks to build production-grade PySpark ETL pipelines that handle messy data, scale efficiently, and run reliably in the cloud.

  Course Objectives ▾

  • Build complete ETL pipelines with proper project structure including extract, transform, and load functions
  • Handle real-world data quality issues including inconsistent formats, test data, and missing values
  • Implement production-standard error handling, logging, and data quality validation
  • Deploy PySpark jobs that run on schedules using spark-submit
  • Diagnose performance bottlenecks using the Spark UI to identify slow operations and inefficient partitioning
  • Apply systematic optimization techniques including caching, partition tuning, and reducing shuffles
  • Understand different managed Spark platforms including Databricks, EMR, and Dataproc
  • Integrate PySpark with cloud storage services like AWS S3 and data catalogs like AWS Glue