The Anatomy of a Search Engine

Sergey Brin, Lawrence Page · Stanford · 1998 · Read the original

The web ranked itself

PageRank treats the web as a directed graph. Each page is a node. Each hyperlink is an edge. A page's importance is determined by the importance of the pages that link to it, recursively. A link from a page that itself has many inbound links counts more than a link from an obscure corner of the web.

The insight is structural: you don't need to understand a page's content to judge its quality. The link structure encodes a distributed, decentralized vote. Millions of authors, each deciding independently what to reference, produce a quality signal that no single editor could replicate. The web's commons generated its own ranking for free.

The algorithm worked because the web in 1998 was still mostly a commons. Links were editorial acts: someone read something, found it valuable, and pointed others to it. PageRank harvested the collective judgment embedded in that structure. It was elegant, effective, and entirely dependent on a commons it did not create.

Appendix A

The technical contribution is in the body of the paper. The prophecy is in the appendix.

"We expect that advertising funded search engines will be inherently biased towards the advertisers and away from the needs of consumers."

Brin and Page wrote this in 1998. They described the exact failure mode of the company they would build. The argument is straightforward: if your revenue comes from advertisers, your incentive is to serve advertisers. The user's interest and the advertiser's interest diverge. An ad-funded search engine will, over time, optimize for the advertiser.

They were right. Google's search results are now a layer of ads above a layer of SEO-optimized content above whatever the user was actually looking for. The authors predicted their own company's corruption before the company existed. The paper is its own exhibit A.

The enclosure

Google didn't just index the commons. It replaced the commons with a proprietary layer. The web's knowledge is still there, but you can't get to it without going through the gate. Search became a toll road on a public highway.

The mechanism was gradual. First, Google was the best way to find things. Then the only way most people found things. Then it started answering questions directly — pulling content from other sites, displaying it in featured snippets. The user got their answer without clicking through. The creator got nothing.

A new technology creates a commons. An intermediary indexes it and becomes indispensable. The intermediary captures the value the commons generated. The commons withers because the incentive to contribute has been redirected. Why publish on the open web when Google will either bury you or extract your content without sending traffic?

A gift that became a fence

PageRank was an academic paper, published openly, describing an algorithm that would create the most valuable advertising company in history. The same minds that understood the web's link structure well enough to exploit it also understood, in Appendix A, that the exploitation was inevitable once advertising entered the picture.

They built it anyway.

The enclosure isn't malice. Network effects in search create a natural monopoly — the more people use one engine, the better its results, the more people use it. The equilibrium is one winner. The original paper warned about exactly this. Paradox of Open Competition