Data Engineering - Apache Airflow
Apache Airflow is a workflow orchestration tool, and primarily used in Big Data related project.
Data Engineering - Apache Airflow free download
Apache Airflow is a workflow orchestration tool, and primarily used in Big Data related project.
Airflow is an extensive tool and has many different features. I will explain in details how we can take benefit from Airflow in our Big Data-related projects. In this Apache Airflow I will start from basic concepts, then move to Installation, Later will deep dive into advanced concepts. In the end, I will create a few big data-related projects and create Airflow Dag to give you a flavor of how all components of Airflow work together.
Please note that Apache Airflow is not a Data Flow tool rather its a Workflow Orchestration tool, With such a nice and beautiful User Interface and a bundle of rich components available. When data is flowing through different processes it is difficult to monitor each and every process, if we are working in CLI based environment. Rather using Apache Airflow we can get notified by color scheming of tasks, and we can start the task right from where it has failed.
I will show you different use cases of Big Data universe, in which we can take leverage from Apache Airflow.
After completing this course Part 1, you will be find your self equipped with a new tool just to solve your Big Data use cases.
