Rethinking Enterprise Search for the Big Data Age Posted on : Sep 01 - 2015

The humble search engine has retained a prominent place in the toolboxes of would-be data explorers. You see Solr and Lucene sitting at the Hadoop table, right alongside SQL and machine learning. But some search experts–including one who helped build Microsoft Bing–says traditional search engines are too long in the tooth for today’s big data challenges.

There’s no doubt that Internet search engines, such as Google and Bing, have changed how we retrieve information, says Donald Thompson, the co-founder and CTO of Maana and a long-time Microsoft developer. But when it comes to the enterprise search engines that companies often use to index and retrieve internal documents, there has been a major failure of innovation.

The problem is that enterprise search has failed to keep up with their consumer-grade cousins, Thompson tells Datanami. “Go to Bing and type in ‘Tom Hanks,'” he says. “You don’t just get Web pages with Tom Hanks. You get the entity experience of Tom Hanks. The search engine knows something about Tom Hanks.”

There’s no such innovation in enterprise search, he says. “There’s not even a page-rank level of innovation,” he adds. “It’s literally just term frequency and documents. They have done some faceting and things like this. But [it’s mostly] a very manual construct. You have to go in, a priori, and define your schemas.” View more