Content Matters

Speaking at O'Reilly's Open Source Convention on Friday, Jimmy Wales provided details on Wikia's plans to enter the search market.

Wales stated that "Internet search is broken".  His biggest complaint is the closed, proprietary nature of the Google algorithms.  Wikia has recently acquired the Grub web crawler technology from Look Smart and plans to combine that with open source search technologies Lucene and Nutch to develop a new open source search engine platform.  Grub is a distributed platform, which can leverage the idle computing power of all those who run it, in order to generate the search index.  In this manner, Wikia can, theoretically, develop a massive search application without the hardware requirements of Google.

Wikia states four guiding principles on its web site:

1. Transparency: Openness in how the systems and algorithms operate, both in the form of open source licenses and open content + APIs.
2. Community: Everyone is able to contribute in some way (as individuals or entire organizations), strong social and community focus.
3. Quality: Significantly improve the relevancy and accuracy of search results and the searching experience.
4. Privacy: Must be protected, do not store or transmit any identifying data.

Wales envisions Wikia to leverage a human-assisted approach, much like Jason Calacanis' recent launch Mahalo, although it sounds like Wikia will use humans in a more limited capacity, for disambiguation of terms.  Jason's initial thoughts on Jimmy's entry into the human-assist search field can be found on his blog.

Never one to shy away from bold statements, Wales suggests that this open source search approach could "shift the power of balance from the search companies back to the publishers".

In my opinion, it's unlikely that anyone will knock Google off its perch in the near-term.  At the same time, the search experience, which had improved dramatically from 1995 - 2002 or so has really stagnated during the past five years, and you could argue that it's gotten worse due to shopping and spam sites and SEO.  New approaches, like those of Wiki and Mahalo, are certainly worth our support and attention.