From zero to Airflow: Bootstrapping into a best-in-class risk analytics platform

To effectively compete in the Fintech space decisions have to be lightning fast and accurate. There is an inherent tradeoff between the two, and to push the needle on both requires a union of excellent models and top notch infrastructure. At BlueVine, we chose to leverage Apache Airflow as an engine for the wide array of models, heuristics and supplemental processes that form our analytics ecosystem. Our key automation tasks (approvals, rejections, fraud assessment) are always underlied by a complex web of dependencies, which is why a workflow design and execution tool such as Airflow is perfect for our needs. But how can a company successfully transition from a legacy system into Airflow, or optimize such a feature packed product so that it actually yields a sufficient return on investment? These are critical and daunting considerations for any company looking to make the switch without incurring any downtime. Indeed, a 2017 McKinsey report found that about 88% of AI initiatives never get deployed despite huge efforts in the experimental stage. In this presentation we will describe the entire process from end-to-end as a case study of how BlueVine got the implementation of Airflow past this hurdle. We will cover: what was planned, which unexpected problems were encountered, and what effect it had on the relationship between the various data teams. We will also detail the mechanism that is in place today, note some real world insights about the strengths and limitations of Airflow, and track several key metrics that were affected by the switch.
Who is this presentation for?
Any member of a data team (data science, data engineering, dev ops) including managers who are considering implementing airflow.
Prerequisite knowledge:
Knowledge with Python and Airflow is recommended but not necessary.
What you'll learn?
Real world insights about the strengths and limitations of Airflow