Back

Speaker "Karthikeyan Ramasamy" Details Back

 

Topic

Twitter Heron - Stream Processing at Scale

Abstract

Storm has long served as the main platform for real-time analytics at Twitter. However, as the scale of data being processed in real- time at Twitter has increased, along with an increase in the diversity and the number of use cases, many limitations of Storm have become apparent. We need a system that scales better, has better debug-ability, has better performance, and is easier to manage – all while working in a shared cluster infrastructure. We considered various alternatives to meet these needs, and in the end concluded that we needed to build a new real-time stream data processing system. This talk will present the design and implementation of the new system, called Heron. Heron is now the de facto stream data processing engine inside Twitter, and share our experiences from running Heron in production.

Profile

Karthik is the engineering manager and technical lead for Real Time Analytics at Twitter. He has two decades of experience working in parallel databases, big data infrastructure and networking. He cofounded Locomatix, a company that specializes in real timestreaming processing on Hadoop and Cassandra using SQL that was acquired by Twitter. Before Locomatix, he had a brief stint with Greenplum where he worked on parallel query scheduling. Greenplum was eventually acquired by EMC for more than $300M. Prior to Greenplum, Karthik was at Juniper Networks where he designed and delivered platforms, protocols, databases and high availability solutions for network routers that are widely deployed in the Internet. Before joining Juniper at University of Wisconsin, he worked extensively in parallel database systems, query processing, scale out technologies, storage engine and online analytical systems. Several of these research were spun as a company later acquired by Teradata. He is the author of several publications, patents and one of the best selling book "Network Routing: Algorithms, Protocols and Architectures." He has a Ph.D. in Computer Science from UW Madison with a focus on databases.