Management for Big Data Applications and Systems

Hadoop has become the very backbone of enterprises working on Big Data. But as Hadoop becomes a critical component of every enterprise’s Big Data needs, its complexity to implement, maintain and develop has increased substantially. Such intricacies create a lot of productivity killers for developers and DevOps personnel. This talk will discuss how enterprises can remove the “Oops” from their Hadoop systems in a reliable, consistent and effective manner. The talk discussions will focus on real-world use cases from the following three key areas: Running Proactive Ops: Running Hadoop operations is all about optimizing workload usage and mitigating risks. The talk will highlight how such operations can be proactively managed, reduce risks and thereby lower the cost of operations. Developing Self-Service for BI users: BI users consistently push the envelopes of any operational Hadoop system. The talk will showcase a self-service system that can arm BI users with application and cluster level information which will go a long way in helping them solve any problems that they may get stuck with. Bringing BI users and Dev Ops Personnel Closer: Professional silos are common in any enterprise. The talk will decipher how such silos can be eliminated by giving BI users and DevOps personnel a single screen/common language to work on complex problems together so that they can solve it quicker with their combined individual expertise - whole is usually more than the sum of individual parts.

Eric Chu is a researcher and software engineer at Unravel. Prior to joining Unravel, he was instrumental in managing a 1500 node, 2000 PB cluster which ran over 4 million Hadoop applications at Rocket Fuel. Before that he designed and implemented Microsoft's first online database management service. This service made it easy for database admins to manage databases on Microsoft SQL Azure. He received his PhD in Computer Science specializing in data management from the University of Wisconsin-Madison.