Crawling where Google dare not crawl

First, the facts:

1. According to Google, Wikipedia.org gets around 28 million unique visitors per DAY
2. According to Hitwise, Google, and other Search Engines send Wikipedia 70% of its total traffic
3. Thus, Search Engines send Wikipedia approximately 18.9 million unique visitors each day!

Now the logic:

Two simple concepts make Wikipedia the site of choice for 19 million searches every day:

1. Search Engines place a lot of weight on “Anchor Text” or the text found linking to a page.
2. The rise of Blogging and Wikipedia Ubiquity

Part 1: Anchor Text

This one is pretty elementary, and begins with Sergei Brin and Lawrence Page’s idea that became Google. They decided to bring order to the web with a “link graph” that would rank pages by popularity.  One of the most useful effects of keeping track of all the links to a page is that one can also look at how other people describe that page.  For example, if a bunch of people link to ebaumsworld.com with the text “this hilarious site” (like this: I LOLed at this hilarious site), Google will decide: “Hey, we should probably show this site to people searching for ‘this hilarious site’.” And they do!

(This is where the Google Bomb comes from, a bunch of people coordinate an effort to link to a page with the exact same anchor text, such as “miserable failure” in the hopes that the page then ranks for that query on Google. Google now prevents this from happening by validating that the text in question also appears somewhere on the site.)

Part 2. The rise of Blogging

Bloggers, for better or worse, have been around since around 1999, but their numbers grew exponentially as blogging platforms such as Blogger and Wordpress became easier and easier to use. Any good blogger will drop multiple links in every post they write.

A few of the most common reasons blogs link are:

1. Credit – another blog or website brought the topic to the blogger’s attention.
2. Definitive Source – If the blogger’s topic clearly has one authoritative source, www.pepsi.com when talking about Pepsi, for example.
3. Reference – If the blogger feels readers might need to know more about a topic, but is too lazy/doesn’t want to explain the topic in post.

This third reason is the one that gives Wikipedia 28 million visitors per day. Blog’s want to send visitors to an impartial accurate, and trustworthy explanation of the topic in question, so they look for the Wikipedia article, and link to it.

Conclusion:

The secret to Wikipedia’s search dominance is in the combination of the two concepts. When a blogger links to a Wikipedia article for reference, they simply highlight the term that needs to be referenced, and create the link. (IE: Then in 1863, came a turning point, the battle of gettysburg)

When linking to Wikipedia, the blogger doesn’t feel he needs to give any disclaimers or additional information about the link, because its leading to a clear, recognizable, and impartial source.

In contrast, if a blogger wanted to link to a non-wikipedia page, they might have to set up the link, describe it in more detail, or make a disclaimer first. (IE: Then in 1863, came a turning point, the Battle of Gettysburg. (See the Army’s interactive Gettysburg page here.)). It’s rare that someone would link to the army page wiith only “Battle of Gettysburg” because frankly there’s more to it then that.