Building a Pipeline Class
In the previous missions of this course on building a data pipeline, we've learned about the concepts of functional programming and how to write Python code using this paradigm. After learning about the functional programming paradigm, we built a sequence of tasks that transformed a raw log file into a summarized CSV file.
When building a data pipeline, you'll want to use a general purpose pipeline that works for all cases as opposed to one specific case. In this mission. We will learn about closures and function decorators that provide additional code re-usability in functional programming. Finally, we will rebuild a static pipeline using the general-purpose pipeline.
Learning how to construct a data pipeline is a critical task for any data engineer. Because it's so critical and difficult to understand how to build a data pipeline without any hands-on practice, you’ll get to apply what you’ve learned from within your browser; there's no need to use your own machine to do the exercises. The Python environment inside of this course includes answer-checking to ensure you've fully mastered each concept before learning the next.
2. Inner Functions
3. Function Closures
4. Python Decorators
5. Method Decorators
6. Decorator Arguments
7. Running the Pipeline
8. Challenge: Making Static Tasks Dynamic
9. Next Steps