The Future Of Big Data Is In The Hybrid Cloud: Part 2 Posted on : Jan 17 - 2019

In part one, I talked about the rise of cloud and big data. Now I want to explore how the two have converged in recent years and why the adoption of big data in hybrid cloud environments will benefit businesses.

Big Data Meets The Cloud

With interest in big data and cloud increasing around the same time, it wasn’t long until big data began being deployed in the cloud. As noted in my previous article, big data comes with some challenges when deployed in traditional, on-premise settings. There’s significant operational complexity, and, worst of all, scaling deployments to meet the continued exponential growth of data is difficult, time-consuming and costly.

The cloud provides the perfect solution to this problem since it was built for convenience and scalability. In the cloud, you don’t have to tinker around trying to manually configure and troubleshoot complicated open-source technology. When it comes to growing your deployments, you can simply hit a few buttons to instantly roll out more instances of Hadoop, Spark, Kafka, Cloudera or any other big data app. This saves money and headaches by eliminating the need to physically grow your infrastructure and then service and manage that larger deployment. Moreover, the cloud allows you to roll back these deployments when you don’t really need them -- a feature that’s ideal for big data’s elastic computing nature.

Big data’s elastic compute requirements mean that organizations will have a great need to process big data at certain times but little need to process it at other times. Consider the major retail players. They likely saw massive surges of traffic on their websites this past Cyber Monday, which generated a reported $7.9 billion in sales. These companies probably use big data platforms to provide real-time recommendations for shoppers as well as to analyze and catalog their actions. In a traditional big data infrastructure, a company would need to deploy physical servers to support this activity. These servers would likely not be needed the other 364 days of the year, resulting in wasted expenditures. However, in the cloud, retail companies could simply spin up the big data and the resources that are needed and then get rid of them when traffic subsides.

This sort of elasticity occurs on a day-to-day basis for many companies that are driving the adoption of big data. Most websites experience a few hours of peak traffic and few hours of light traffic each day. Think of social media, video streaming or dating sites. Elasticity is a major feature of big data, and the cloud provides the elasticity to keep those sites performing under any conditions. View More