As businesses are realizing the power Hadoop and large data analytics, many businesses are demanding large scale real time streaming data analytics. Apache Storm and Apache Spark are platforms that can process large amount of data in real time. However building applications on these platforms that can scale, reliably process data without any loss, satisfy functional needs and at the same time meet the strict latency requirements, takes lot of work to get it right.
After implementing multiple large real time data processing applications using these technologies in various business domains, we distilled commonly required solutions into generalized design patterns. These patterns are proven in the very large production deployments where they process millions of events per second, tens of billions of events per day and tens of terabytes of data per day.

Latency sensitive lossless micro batching, high scale data enrichment through external systems lookup, dynamic rules and alerts, adaptive self tuning, real time stream joins are some of patterns to name.

This talk covers these proven design patterns and for every design pattern it covers – problem statement, applicability of design pattern, the pattern design and sample code demonstrating the implementation.

Attendees can take advantages of these patterns in building their applications and improve their productivity, quality of solution as well as success factor of their applications.