Why Big Data Projects Fail and How to Avoid It Posted on : Nov 04 - 2016

More data has been created in the past two years than in the entire previous history of the human race. In digital business, data about an organization’s customers, competition, internal operations and more is being generated at an unprecedented rate. There’s enormous value to be unlocked – for a typical Fortune 1000 company, a 10% increase in data accessibility will result in more than $65 million additional net income.

Big data analytics enables organizations to leverage the right data to accurately establish trends and predict future outcomes. As every industry seeks the benefits of data-driven intelligence, IDC forecasts an enormous $50 billion spend on big data solutions and services by 2019.

Through big data analytics, businesses can understand trends and future outcomes around consumer purchase behavior and sentiment. Take customer churn for example. Businesses want to minimize customers departing or transitioning to a less profitable product. This insight is crucial to taking tailored action – be it driving revenue through personalized offers or creating programs for existing customers to reward loyalty – that will improve customer experience and ultimately increase share of wallet. Other insight drives action towards enhancing products, reducing risk, improving performance and enhancing operations.

Automation is a requirement of successfully operationalizing a cost-effective and long-term big data strategy. Yet 60 percent of big data projects will fail to go beyond piloting and will be abandoned, according to Gartner.

A critical success factor to realizing big data’s benefits is how you operationalize big data projects. The main considerations are:

1 — Optimizing existing infrastructure

Exponential data growth poses a challenge to business infrastructure. Businesses will need to know “How much capacity do I have?”, “How much capacity do I need?” and “How am I going to grow the infrastructure in line with the business needs?”

Predictive modelling enables CIOs to forecast future usage and predict infrastructure growth.

The right tools must also be used to optimize an organization’s existing infrastructure. Enterprise grade capacity optimization and capacity visualization are two values that the right IT operations management tool can bring.

2 — Achieving speed and scale

Getting data sets from multiple sources (such as ERP, CRM, social media, devices and sensors) into a big data platform manually would be extremely time- and labor-intensive, involving integration of multiple tools and technologies. The right automation tool enables data to flow in seamlessly, essential to repeat the success of a pilot big data use case across multiple cases.

However, an automation solution requiring a lot of custom development can be restrictive in achieving scale and speed. It must be able to plug seamlessly into a company’s existing technology ecosystem.

The data supply chain must also be considered in its entirety. Deploying separate tools to ingest data, process it and extract it for analysis creates silos. An end-to-end solution sets big data projects up for success.

3 — Computing the right data at the right time

Businesses need to glean the right insights to impact the bottom line from the increasing big data lake. To successfully accelerate insights, the big data technology ecosystem must interface with other enterprise applications and data sources, such as ERP solutions and connected devices.

In addition to managing the data workflows end-to-end, IT must also manage the deadlines for those workflows to ensure analytics teams can view ‘on-time’ data.

4 — Protecting data with visibility

If organizations hesitate to invest in big data, it’s usually due to security concerns. No company wants to fall victim to the next data breach.

You can only protect what you know about. In the case of big data, the businesses that get security right have holistic visibility of how the big data infrastructure is connected to enterprise applications. This is vital to take measures such as managing access to sensitive data that’s being moved or stored.

Becoming data-driven can involve navigating uncharted territory to make the significant technology and cultural changes required. But a strategic approach to managing, sustaining and growing a big data environment, involving the right technologies, means organizations can overcome operational roadblocks to focus on insights that help their businesses thrive in the digital era. Source