Search engines today do what they currently do 100%!

Image representing Google as depicted in Crunc...Image via CrunchBase, source unknown Mike Arrington is rightly dumbfounded to hear that search is 90% done. Marissa Mayer said as a much to the LA Times.

From one perspective–where you look at search and the solution to it as being indexing document and making statistical connections from queries to documents–perhaps the game nearly is done. In other words, if you measure the success of search engines against the yardstick of what they already do, then I guess they are there.

In the same way that the grand ocean liners were the pinnacle, nay the 95% done of transatlantic travel, until, er, the jet age. Or that Bessie, my trusty shire horse, was the best way of getting from London to Windsor on a rainy day.

This is setting up for a classic Clayton Christensen disruption where the way in which the problem is framed limits the way in which resources are allocated to crack the problem (from an R&D stand point) and then suffocates it (from a commercialisation standpoint).

Forgetting for a moment what other companies are trying, start from the stand point of the end user.

I have been at internet user for seventeen years (nearly half my life), I remember the days of Archie, Veronica, Gopher (but no Jughead). And having worked with two search companies, even today I will struggle with certain types of search queries:

Remember a search query is not the thing that Google/Altavista/Webcrawler does–which is to try to statistically match a string of words in a query to similar words in a document. A search query is a request for information, it is me as a human (or me as a cyborg or sentient machine) looking to find something out. The seach query–the bit I type in to the search bar or shout at my assistant is merely my attempt to communicate the representation of the request I have in my head.

And frankly there is no way that search is 90% solved when it comes to that customer problem. Just today, things I was looking for but couldn’t find.

  • I need to find someone to come round to my house and fix my bike. There I said it. But I sure as hell haven’t been able to Google, Gumtree or Craigslist it.
  • I can’t get Xgrid Controller to start on Leopard. I can google it
    but I can’t get anything useful. I know someone has some wisdom about this but where?
  • I need to find some cool dresses for my daughter’s first birthday party (for her, not me), and frankly the unthrilling prospect of an hours iterative searching through unfamiliar fashion blogs and kids clothes stores does not seem like a solution.
  • I wanted to find out the nutritional value of the food I ate at a party today for my food log. Randomly catered food, strange portions? Nadda di nadda from Google

So from the perspective of solving the problem that search engines currently solve today, search is pretty much solved, better than 90%. But who are we kidding? What they currently solve today is some weird, sub-human information retrieval problem.

Yes. It is better than Z39.50 searching.

But let’s not get confused the relief that it isn’t completely atrocious with the reality that we just aren’t done yet. Learned helplessness is a horrible thing, and ten minutes observing someone trying to solve some real problems will find a large class of them is just unresolvable on today’s Internet.

But they will be resolved.

Who is interesting in this space?
True Knowledge (usual disclosures apply) : totally unique approach to solving the fact-oriented research problem. (Ping me on Twitter if you want a beta, give me your email address)

  • Freebase is another good example but has yet to crack the hard bit of figuring out what a user’s intent is and how to translate that to a query
  • Twitter has potential, because you can fling your request into the information stream and sometimes get a human to come back with you. Twine might fall into the same category. Although I question whether Twitter can scale enough to handle this and whether we’ll ever understand Twine.

    The raft of semantic layer companies out there will no doubt help. Vertical search can also help, although someone has to help me find the right vertical search engine. I doubt whether the visualisation guys are close to changing the paradigm although they might be a nice bit of polish.

    In the meantime, I have four queries up top if anyone wants to help me with them…

  • Reblog this post [with Zemanta]


    1. Posted September 9, 2008 at 8:14 am | Permalink

      This is a very good article. Sounds to me as if you should be interested in ‘expert search’.

      If you could find an expert in the four areas that interest you (above), then you could easily get the answers to your questions in no time.

      Maybe you should look for/found a social network (with search functionality) of ‘experts’ who are willing to help (for free, because they love a particular subject area?

    2. Posted September 9, 2008 at 8:20 am | Permalink

      Yes–but the costs of finding the right expert of expert search system are extremely high. I use online personal assistant for a lot of this but I am literally paying a human to do the search for me.

      As for the query on XGrid I have fortunately been able to escalate to someone in Apple in Cupertino. But several days later, still no answer.

      As for the fourth request, good luck!

    3. Posted September 10, 2008 at 11:41 am | Permalink

      Sorry, my language wasn’t that clear. Let me try again.

      Imagine there was a start-up that was some kind of social network/search company where one could search for experts by keyword.

      For example, you could search for XGrid and 3 people would come up. You could then contact one (or several) and ask your question.

      Imagine it like an answers but with self-declared specialists.

      I could then also search through that knowledge base and find answers or specialists.

      I guess Mahalo may be going in that direction.

    One Trackback

    1. By Answers vs search « azeem.azhar on October 6, 2008 at 10:40 am

      […] line in innovations they are bringing to the search space. I am a big believer that there is lots more innovation to come in search. In fact, I am deeply involved in True Knowledge which is an open-domain question answering system […]

    Post a Comment

    You must be logged in to post a comment.

    %d bloggers like this: