Building a Pipeline Class

In the previous lessons of this course on building a data pipeline, we’ve learned about the concepts of functional programming and how to write Python code using this paradigm. After learning about the functional programming paradigm, we built a sequence of tasks that transformed a raw log file into a summarized CSV file.

When building a data pipeline, you’ll want to use a general purpose pipeline that works for all cases as opposed to one specific case. In this lesson. We will learn about closures and function decorators that provide additional code re-usability in functional programming. Finally, we will rebuild a static pipeline using the general-purpose pipeline.

Learning how to construct a data pipeline is a critical task for any data engineer. Because it’s so critical and difficult to understand how to build a data pipeline without any hands-on practice, you’ll get to apply what you’ve learned from within your browser; there’s no need to use your own machine to do the exercises. The Python environment inside of this course includes answer-checking to ensure you’ve fully mastered each concept before learning the next.

Objectives

  • Learn to use functional closures in Python.
  • Learn to write decorators and apply them to functions.
  • Implement a well desgined pipeline API.

Lesson Outline

  1. Overview
  2. Inner Functions
  3. Function Closures
  4. Python Decorators
  5. Method Decorators
  6. Decorator Arguments
  7. Running the Pipeline
  8. Challenge: Making Static Tasks Dynamic
  9. Next Steps
  10. Takeaways

Get started for free

No credit card required.

Or With

By creating an account you agree to accept our terms of use and privacy policy.