Data Processing and Kubernetes

Kubernetes is a fast growing open-source platform which provides container-centric infrastructure. Conceived by Google in 2014, and leveraging over a decade of experience running containers at scale internally, it is one of the fastest moving projects on GitHub with 1000+ contributors and 40,000+ commits. Kubernetes has first class support on Google Cloud Platform, Amazon Web Services, and Microsoft Azure. Kubernetes started as a general purpose orchestration framework with a focus on serving jobs. Since then, more and more classes of workloads and use-cases are moving in to Kubernetes, ranging from stateful applications like databases, file systems and message queues, to data processing frameworks like Spark. This talk will focus on some Kubernetes features and constructs being worked on by the open source community to support all these classes of workloads.