Search engines today do what they currently do 100%!

Image via CrunchBase, source unknown Mike Arrington is rightly dumbfounded to hear that search is 90% done. Marissa Mayer said as a much to the LA Times.

From one perspective–where you look at search and the solution to it as being indexing document and making statistical connections from queries to documents–perhaps the game nearly is done. In other words, if you measure the success of search engines against the yardstick of what they already do, then I guess they are there.

In the same way that the grand ocean liners were the pinnacle, nay the 95% done of transatlantic travel, until, er, the jet age. Or that Bessie, my trusty shire horse, was the best way of getting from London to Windsor on a rainy day.

This is setting up for a classic Clayton Christensen disruption where the way in which the problem is framed limits the way in which resources are allocated to crack the problem (from an R&D stand point) and then suffocates it (from a commercialisation standpoint).

Forgetting for a moment what other companies are trying, start from the stand point of the end user.

I have been at internet user for seventeen years (nearly half my life), I remember the days of Archie, Veronica, Gopher (but no Jughead). And having worked with two search companies, even today I will struggle with certain types of search queries:

Remember a search query is not the thing that Google/Altavista/Webcrawler does–which is to try to statistically match a string of words in a query to similar words in a document. A search query is a request for information, it is me as a human (or me as a cyborg or sentient machine) looking to find something out. The seach query–the bit I type in to the search bar or shout at my assistant is merely my attempt to communicate the representation of the request I have in my head.

And frankly there is no way that search is 90% solved when it comes to that customer problem. Just today, things I was looking for but couldn’t find.

• I need to find someone to come round to my house and fix my bike. There I said it. But I sure as hell haven’t been able to Google, Gumtree or Craigslist it.
• I can’t get Xgrid Controller to start on Leopard. I can google it
  but I can’t get anything useful. I know someone has some wisdom about this but where?
• I need to find some cool dresses for my daughter’s first birthday party (for her, not me), and frankly the unthrilling prospect of an hours iterative searching through unfamiliar fashion blogs and kids clothes stores does not seem like a solution.
• I wanted to find out the nutritional value of the food I ate at a party today for my food log. Randomly catered food, strange portions? Nadda di nadda from Google

So from the perspective of solving the problem that search engines currently solve today, search is pretty much solved, better than 90%. But who are we kidding? What they currently solve today is some weird, sub-human information retrieval problem.

Yes. It is better than Z39.50 searching.

But let’s not get confused the relief that it isn’t completely atrocious with the reality that we just aren’t done yet. Learned helplessness is a horrible thing, and ten minutes observing someone trying to solve some real problems will find a large class of them is just unresolvable on today’s Internet.

But they will be resolved.

Who is interesting in this space?
True Knowledge (usual disclosures apply) : totally unique approach to solving the fact-oriented research problem. (Ping me on Twitter if you want a beta, give me your email address)

Freebase is another good example but has yet to crack the hard bit of figuring out what a user’s intent is and how to translate that to a query

Twitter has potential, because you can fling your request into the information stream and sometimes get a human to come back with you. Twine might fall into the same category. Although I question whether Twitter can scale enough to handle this and whether we’ll ever understand Twine.

The raft of semantic layer companies out there will no doubt help. Vertical search can also help, although someone has to help me find the right vertical search engine. I doubt whether the visualisation guys are close to changing the paradigm although they might be a nice bit of polish.

In the meantime, I have four queries up top if anyone wants to help me with them…