Accelerated Spark on Azure: Seamless and Scalable Hardware Offloads in the Cloud

Microsoft Azure’s advanced compute and network infrastructure allows Spark to run in the cloud without compromising on performance. With the growing arsenal of hardware offloads available on cloud VMs, owning and maintaining bleeding edge hardware is no longer a prerequisite for accelerated compute. In this talk, we will demonstrate how hardware accelerations in Azure can be utilized to speed-up Spark jobs seamlessly, with the aid of RDMA (Remote Direct Memory Access) support in the VM. We will demonstrate use cases of benchmarks and real-world applications, that achieve impressive performance improvements with minimal configuration.