In the previous lessons of this course on building a data pipeline, we've learned about the concepts of functional programming and how to write Python code using this paradigm. After learning about the functional programming paradigm, we built a sequence of tasks that transformed a raw log file into a summarized CSV file.

When building a data pipeline, you'll want to use a general purpose pipeline that works for all cases as opposed to one specific case. In this lesson. We will learn about closures and function decorators that provide additional code re-usability in functional programming. Finally, we will rebuild a static pipeline using the general-purpose pipeline.

Learning how to construct a data pipeline is a critical task for any data engineer. Because it's so critical and difficult to understand how to build a data pipeline without any hands-on practice, you’ll get to apply what you’ve learned from within your browser; there's no need to use your own machine to do the exercises. The Python environment inside of this course includes answer-checking to ensure you've fully mastered each concept before learning the next.

Objectives

  • Learn to use functional closures in Python.
  • Learn to write decorators and apply them to functions.
  • Implement a well desgined pipeline API.

Lesson Outline

1. Overview
2. Inner Functions
3. Function Closures
4. Python Decorators
5. Method Decorators
6. Decorator Arguments
7. Running the Pipeline
8. Challenge: Making Static Tasks Dynamic
9. Next Steps
10. Takeaways