Airflow use of DAG to model workflows
https://airflow.apache.org/concepts.html#dags
This is a link to a documentation page of Airflow Platform. Airflow, developed by Airbnb, allows implementation of workflows using directed acyclic graphs. The platform also allows execution of the implemented workflows as well as monitoring. This project interested me because of its growing use cases. As companies become larger and technologically advanced, there is a growing need for a streamlined and scalable way to implement many of their practices. Large industrial processes can be modeled using DAG’s in the Airflow platform which speeds up the overall workflow of the company. For instance, Airflow can be used to automate the extracting, transforming and loading of a company’s daily data. By executing the ETL pipeline before the workday begins, the most recent data ready for use by the different teams within the company is available at the start of the workday. This would save time for all the teams using the data, speeding up the overall workflow of the company.
Airflow models workflows using digital acyclic graphs or dependency networks shown in class. In a DAG or dependency network, the nodes are different tasks with the directed edges showing the dependencies between the tasks. While the network from class shows a flowchart of college courses with each node/task being a course, Airflow supports many different operations for a node/task. Support for bash commands, python functions, emails, HTTP requests and SQL commands as well as many other command business operations allows for the airflow to model a wide variety of industrial processes.