Achieving data science productivity and scale through an enterprise Signal Layer

“Today, there is tremendous demand to infuse data science and advanced analytics into diverse use cases throughout the enterprise – from customer service, promotion and acquisition, to pricing, operations and optimization. However, these projects are typically developed “one use case at a time” whereby a team of data scientists gets the request, proceeds to find the right data, cleaning and preparing variables, training and testing models, and eventually packages the developed code for production. This process is highly inefficient, as data preparation steps are often repeated within each use case, multi-functional (hard to find) experts are needed, and insights get trapped into the code, with limited reusability. We created an approach that breaks the productivity bottleneck by focusing on building a “Signal Layer” – a collection of reusable, highly processed analytical objects, both descriptive and predictive, that power multiple use cases, and are organized around several key entities (such as customers and products). We use examples from large Hadoop-native enterprise deployments in Retail, Travel, Telecom and other industries. We will show how, after the signal layer was built, new use case creation went from months to weeks”