What you missed in Big Data: Moving beyond science project Posted on : Jan 28 - 2015

The analytics ecosystem marked several major milestones along the journey to turn data insights into enterprise reality last week as vendors tore down more of the barriers holding back organizations from unlocking the value of their growing information troves. But it’s the Apache Software Foundation that made the biggest advance after inaugurating Falcon as a top-level project.

The framework provides a high-level interface for defining automated controls to mange the flow of data in Hadoop, a task that becomes practically impossible to handle manually on the scale of typical production clusters. The technology enables engineers to determine at a granular level what to do with a piece of information in a particular scenario, such as when a user tries to access a piece of information they’re not authorized to access or a sensitive file is modified.

That functionality makes it feasible for everyday organizations to enforce governance codes and meet regulatory compliance across terabytes or petabytes of data stored in their analytics clusters, making Hadoop that much more viable in enterprise settings. However, upholding information management policies is not a challenge that is limited to the batch processing engine.

Ensuring everything is in check with privacy requirements is also difficult when it comes to the legacy silos where organizations store most of their mission-critical data, which are typically so numerous that practitioners have no way of chasing down potential inconsistencies efficiently. Like with Hadoop, automation through analytics is the answer – at least according to Dell, which followed up Falcon’s inauguration with a new version of its infrastructure monitoring software that takes direct aim at compliance.

InTrust 11.0 introduces what the company describes as a search engine for operations professionals that makes it possible to quickly scan relationships between applications and data for policy breaches. Security professionals can also take advantage of that functionality to identify correlations between compromised files and user activity.

But as valuable as it is in the IT department, the ability to quickly drill down into specific subsets of information without having to wade through a mountain of unrelated statistics holds even greater potential for other parts of the organization. Marketers in particular stand to gain from that kind of visibility when handling advertising data, one of the most complex workloads around, but there aren’t many solutions that provide such functionality in an easily accessible way.

Metamarkets Inc. hopes to fill the gap with Facet, a new visualization engine for its database that – true to its name – makes it possible to slice and dice information without the need for the complex queries required to accomplish similar results in traditional solutions. Introduced last Wednesday, the software exposes common commands as interface elements that marketers can apply to their campaign data in chains of up to five.  Source