Building a Pipeline Class

In the previous missions of this course on building a data pipeline, we've learned about the concepts of functional programming and how to write Python code using this paradigm. After learning about the functional programming paradigm, we built a sequence of tasks that transformed a raw log file into a summarized CSV file.

When building a data pipeline, you'll want to use a general purpose pipeline that works for all cases as opposed to one specific case. In this mission. We will learn about closures and function decorators that provide additional code re-usability in functional programming. Finally, we will rebuild a static pipeline using the general-purpose pipeline.

Learning how to construct a data pipeline is a critical task for any data engineer. Because it's so critical and difficult to understand how to build a data pipeline without any hands-on practice, you’ll get to apply what you’ve learned from within your browser; there's no need to use your own machine to do the exercises. The Python environment inside of this course includes answer-checking to ensure you've fully mastered each concept before learning the next.


  • Learn to use functional closures in Python.
  • Learn to write decorators and apply them to functions.
  • Implement a well desgined pipeline API.

Mission Outline

1. Overview
2. Inner Functions
3. Function Closures
4. Python Decorators
5. Method Decorators
6. Decorator Arguments
7. Running the Pipeline
8. Challenge: Making Static Tasks Dynamic
9. Next Steps
10. Takeaways


Course Info:


The median completion time for this course is 6.3 hours. ​View Details​​​

This course requires a premium subscription. This course has four missions, and one guided project.  It is the seventh course in the Data Engineer Path.


Take a Look Inside

(function(d) { d.addEventListener("DOMContentLoaded", function() { var pathname = d.location.pathname.replace(/^[/]|[/]$/g, "").replace("/", "-"); var tags = d.getElementsByTagName("iframe"); var type = pathname.startsWith("course") ? "?course=" : pathname.startsWith("path") ? "?path=" : null; if (type) { var i; for (i = 0; i < tags.length; i++) { if (tags[i].src.indexOf("signup#iframe") !== -1) { tags[i].src = tags[i].src.replace("#iframe", "") + type + pathname + "#iframe"; } } } }, false); })(document);