Building a Pipeline Class
In the previous lessons of this course on building a data pipeline, we’ve learned about the concepts of functional programming and how to write Python code using this paradigm. After learning about the functional programming paradigm, we built a sequence of tasks that transformed a raw log file into a summarized CSV file.
When building a data pipeline, you’ll want to use a general purpose pipeline that works for all cases as opposed to one specific case. In this lesson. We will learn about closures and function decorators that provide additional code re-usability in functional programming. Finally, we will rebuild a static pipeline using the general-purpose pipeline.
Learning how to construct a data pipeline is a critical task for any data engineer. Because it’s so critical and difficult to understand how to build a data pipeline without any hands-on practice, you’ll get to apply what you’ve learned from within your browser; there’s no need to use your own machine to do the exercises. The Python environment inside of this course includes answer-checking to ensure you’ve fully mastered each concept before learning the next.
- Learn to use functional closures in Python.
- Learn to write decorators and apply them to functions.
- Implement a well desgined pipeline API.
- Inner Functions
- Function Closures
- Python Decorators
- Method Decorators
- Decorator Arguments
- Running the Pipeline
- Challenge: Making Static Tasks Dynamic
- Next Steps