Speaker "Maxime Beauchemin" Details Back
-
Name
Maxime Beauchemin
-
Company
Airbnb
-
Designation
Data Engineer
Topic
Airflow - An open source workflow engine
Abstract
Airflow - An open source workflow engine Company DAGs (directed acyclic graphs) of jobs are evolving rapidly, and orchestrating and monitoring jobs inside and outside of Hadoop has been a growing challenge for all data driven companies. Airflow is a new platform developed at Airbnb that allows data workers to programmatically author data pipelines. It offers an extensible set of operators that can be used to orchestrate workflows across many data systems as well as a rich user interface to monitor and troubleshoot complex DAGs. Airflow is the future of complex batch processing for companies with complex analysis needs.