A large-scale experiment created over 2 billion web pages, linking to each other to analyse the patterns the various search engines crawled the site. Each page contained a link to two new pages, as well as a parent, and the three main search engines were watched as they scanned the site.
Very cool…
