Winning at Search: The Algorithm or The Infrastructure?

We are on the eve of Google announcing their search results for their 3Q. Google has become a major force in discovery and advertising by virtue of their ability to surface the closest result relevant to a user across the broadest set of queries on the Internet. Dozens of start-ups and certainly a few large players have tried to de-throne Google’s supremacy, but few have been successful. The switching costs are zero, yet Google’s market share has only gone up. Narrowing the domain has helped, and by limiting topical areas to things like shopping or health, companies have created market share distributions more favorable than in broad search; however, an end user is not going to use or remember 100 different search engines optimized for 100 different topics. In fact, as it has in Health or in Local, Google has picked off verticals one by one to super-optimize. This all got me thinking about how a start-up could ever beat Google at the broad game of search.

Search is decomposed into a few different elements. The first is a “spider” – a virtual bot that scours the web, parses web pages, and builds a representation of the web; the second is an algorithm that takes those spliced pieces and decides what pages are more important than others given a set of constraints or inputs; the third is a massive index that takes all this analysis and stores it so that at “query time”, an engine can quickly take the digested knowledge and weights, and return a result.

It’s my view that algorithms are not people or resource intensive. A few guys thinking very hard can come up with simple, revolutionary ideas as Sergey Brin and Larry Page did. Sure, Google has an incredible number of variables and residual terms that help refine its algorithm, but at the end of the day, it’s very rare that math is invented or discovered. In fact, I’d wager a “better algorithm” already exists somewhere in academic labs throughout the country. If it can be written or built by few, it is within the realm of startup possibility today.

I tend to believe the biggest challenge for a start-up remains circumventing the need to re-create Google’s infrastructure against an algorithm. Google spends over $2.8bln in CAPEX a year. They spend significantly more in CAPEX than they do on search algorithm specific R&D. I have heard estimates that maintenance and improvement of Google’s algorithms can be satisfied by a few hundred engineers.