MISSION 265

Building a Pipeline Class

In the previous missions of this course on building a data pipeline, we've learned about the concepts of functional programming and how to write Python code using this paradigm. After learning about the functional programming paradigm, we built a sequence of tasks that transformed a raw log file into a summarized CSV file.

When building a data pipeline, you'll want to use a general purpose pipeline that works for all cases as opposed to one specific case. In this mission. We will learn about closures and function decorators that provide additional code re-usability in functional programming. Finally, we will rebuild a static pipeline using the general-purpose pipeline.

Learning how to construct a data pipeline is a critical task for any data engineer. Because it's so critical and difficult to understand how to build a data pipeline without any hands-on practice, you’ll get to apply what you’ve learned from within your browser; there's no need to use your own machine to do the exercises. The Python environment inside of this course includes answer-checking to ensure you've fully mastered each concept before learning the next.

Objectives

  • Learn to use functional closures in Python.
  • Learn to write decorators and apply them to functions.
  • Implement a well desgined pipeline API.

Mission Outline

1. Overview
2. Inner Functions
3. Function Closures
4. Python Decorators
5. Method Decorators
6. Decorator Arguments
7. Running the Pipeline
8. Challenge: Making Static Tasks Dynamic
9. Next Steps
10. Takeaways

building-a-data-pipeline

Course Info:

Intermediate

The median completion time for this course is 6.3 hours. ​View Details​​​

This course requires a premium subscription. This course has four missions, and one guided project.  It is the seventh course in the Data Engineer Path.

START LEARNING FREE

Take a Look Inside