Course

Deploying to the Cloud

You’ve learned cloud fundamentals and built a data pipeline locally with Docker and Airflow. Now it’s time to deploy. But cloud platforms aren’t interchangeable — AWS and GCP organize resources differently, use different services, and follow different patterns. This course gives you hands-on deployment experience with both. You’ll deploy a complete Apache Airflow pipeline to AWS using ECS, Fargate, S3, RDS, and an Application Load Balancer. Then you’ll take the same pipeline and deploy it to GCP using a Compute Engine VM and Cloud Storage — a simpler architecture that shows how the same Docker and Airflow skills transfer across platforms. By the end, you’ll have deployed production-grade pipelines to both AWS and GCP, giving you the flexibility to work with whichever platform your team uses.

Enroll for free

Part of the Data Engineer (Python) path.

4.8 (359 reviews)

40 learners enrolled in this course.

Intermediate friendly
10 hours
Self paced
5 lessons

“The learning paths on Dataquest are incredible. They give you a direction through the learning process – you don’t have to guess what to learn next.”

Otávio Silveira

Data Analyst @ Hortifruti

Deploying to the Cloud

Enroll For Free

Course overview

You understand cloud fundamentals — now deploy for real. This course takes you through hands-on deployment to both AWS and GCP. On AWS, you’ll provision S3 buckets, RDS databases, and deploy Airflow to ECS with Fargate — a managed container architecture with load balancing and auto-scaling. On GCP, you’ll take the same pipeline and deploy it to a Compute Engine VM with Docker Compose, configure service accounts for secure access, and upload pipeline output to Cloud Storage. You’ll see how the same Airflow and Docker skills transfer across platforms while the infrastructure layer changes. By the end, you’ll have deployed working pipelines to both major cloud providers — giving you the versatility to work on any team, with any stack.

Key skills

Provisioning AWS infrastructure including storage, databases, and networking resources
Deploying containerized applications to AWS using ECS and Fargate
Building production-grade cloud orchestration systems that scale automatically
Adapting Dockerized pipelines across cloud platforms by swapping provider-specific components
Deploying Airflow to GCP using Compute Engine with Docker Compose
Configuring GCP service accounts and Cloud Storage for secure, automated data delivery

Course outline

Deploying to the Cloud [5 lessons]

Deploying Airflow to the Cloud with Amazon ECS (Part I) 2h

Lesson Objectives

Understand the limitations of local ETL scripts
Set up Apache Airflow locally using Docker Compose
Configure Airflow environment variables and execution settings
Create and deploy a DAG with Python operators
Define task dependencies and execution order in workflows

Deploying Airflow to the Cloud with Amazon ECS (Part II) 2h

Lesson Objectives

Set up AWS S3 bucket for pipeline output storage
Create and configure IAM roles for secure ECS-S3 communication
Configure security groups and Application Load Balancer for service access
Deploy PostgreSQL RDS database for Airflow metadata persistence
Test end-to-end DAG execution with local-to-cloud S3 uploads

Deploying Airflow to the Cloud with Amazon ECS (Part II) 2h

Lesson Objectives

Build and push custom Airflow Docker images to ECR
Configure ECS task definitions with environment variables and commands
Deploy Airflow components using ECS Fargate services and tasks
Troubleshoot common Airflow deployment issues on AWS ECS
Understand and optimize AWS costs for cloud-hosted Airflow

Deploying Airflow to the Cloud with Google Cloud Platform (Part I) 2h

Lesson Objectives

Adapt an existing Airflow project for GCP deployment
Provision and configure a Compute Engine VM instance
Deploy containerized Airflow using Docker Compose on GCP
Secure cloud infrastructure with firewall rules and static IPs
Troubleshoot common Airflow deployment issues on cloud VMs

Deploying Airflow to the Cloud with Google Cloud Platform (Part II) 2h

Lesson Objectives

Understand how Google Cloud Storage provides persistent data storage
Create and configure GCS buckets with appropriate regional settings
Set up service accounts for secure container authentication
Implement automatic credential discovery using GCP metadata server
Build complete ETL pipelines that upload to cloud storage

The Dataquest guarantee

Dataquest has helped thousands of people start new careers in data. If you put in the work and follow our path, you’ll master data skills and grow your career.

We believe so strongly in our paths that we offer a full satisfaction guarantee. If you complete a career path on Dataquest and aren’t satisfied with your outcome, we’ll give you a refund.

Master skills faster with Dataquest

Go from zero to job-ready

Learn exactly what you need to achieve your goal. Don’t waste time on unrelated lessons.

Build your project portfolio

Build confidence with our in-depth projects, and show off your data skills.

Challenge yourself with exercises

Work with real data from day one with interactive lessons and hands-on exercises.

Showcase your path certification

Share the evidence of your hard work with your network and potential employers.

Grow your career with
Dataquest.

98%

of learners recommend

Dataquest for career advancement

4.85

Dataquest rating

SwitchUp Best Bootcamps

$30k

Average salary boost

for learners who complete a path

Aaron Melton

Business Analyst at Aditi Consulting

“Dataquest starts at the most basic level, so a beginner can understand the concepts. I tried learning to code before, using Codecademy and Coursera. I struggled because I had no background in coding, and I was spending a lot of time Googling. Dataquest helped me actually learn.”

Jessica Ko

Machine Learning Engineer at Twitter

“I liked the interactive environment on Dataquest. The material was clear and well organized. I spent more time practicing then watching videos and it made me want to keep learning.”

Victoria E. Guzik

Associate Data Scientist at Callisto Media

“I really love learning on Dataquest. I looked into a couple of other options and I found that they were much too handhold-y and fill in the blank relative to Dataquest’s method. The projects on Dataquest were key to getting my job. I doubled my income!”