Multiple Dependency Pipeline
At the end of the previous lesson on building a pipeline class, we discussed some drawbacks with our initial pipeline implementation. One of the drawbacks was the restriction of only linear running tasks. Using our tasks as an example, we’ll show why this is a major drawback that we must address when we’re building a data pipeline.
In our last lesson’s task pipeline, the final task was to summarize logs outputted from a parsed CSV file. But suppose we wanted to also run a summarize on multiple column. This seems doable — our only requirement should be the parsed CSV — but with our linear pipeline, this will not work.
In this lesson, we will solve the linear dependency mapping problem of the pipeline created in the previous lesson. We will discuss the DAG or directed acrylic graph, and how we can use it as a task scheduler. Then, we will implement the DAG, and add it to our pipeline.
As you get familiar with the concepts in these multiple dependency pipelines, you’ll get to apply what you’ve learned from within your browser; there’s no need to use your own machine to do the exercises. The Python environment inside of this course includes answer-checking to ensure you’ve fully mastered each concept before learning the next.
- Learn the basics of graph theory.
- Learn to implement a directed acyclic graph in Python.
- Write a schduler for the pipeline class.
- Intro to DAGs
- The DAG Class
- Sorting the DAG
- Finding Number of In Degrees
- Challenge: Sorting Dependencies
- Enhance the Add Method
- Adding DAG to the Pipeline
- Challenge: Running the Pipeline
- Next Steps