What’s the Problem?

Do a quick Google search for Brown Bailout and take a look at the first few results.

Notice how BrownBailout.com's organic listing has no description.

It’s  easy to miss the first organic result, the link to the official webpage Brownbailout.com. The listing has only a basic two-word title, and no-description or cache links.  Not only that, but a search for site:brownbailout.com (to reveal all the pages within the site) turns up only that page, and nothing more!

Why did it happen?

The simplest SEO mistake in the book. Literally one character is costing BrownBailout.com thousands of visitors and advertising dollars. The culprit can be found here: http://www.brownbailout.com/robots.txt and looks like this:

User-agent: *
Disallow: /

This robots.txt is a little text file that sits on the server telling Search Engines “Stay Away!”  Typically this is placed on sites or sections of sites that have content that is expired, redundant or inferior. I am guessing someone from the PR or Advertising Agency that created BrownBailout.com left it up by mistake.  Sometimes misinformed developers or designers use the “User-agent: * Disallow: /” code as a way of soft-launching the site for a private group of users to see. This is very bad procedure for two reasons:

1. Not Secure: Anyone can type in the address and find the “private” site. Often times newly purchased domains come up on certain site’s radars automatically, creating links to the private site.
2. Easy to forget about: With the myriad other steps to worry about during a launch, a simple hidden txt file is very easy to overlook.

What are the effects?

Here is what the Google results page would look like if they fixed the robots.txt:

BrownBailout.com could own way more results page real estate if they fixed their SEO problem.

This would cause two major improvements:

1. Increase CTR on BrownBailout.com with more links and more real estate – that’s a no-brainer
2. Decrease CTR on negative articles and UPS paid ad – the extra sitelinks for BrownBailout.com attract a users attention away from the negative UPS ad, and push down the negative organic search result that is currently in third position.

How to fix it?

Come on dummy, take down the robots.txt file! Or even better, just TAKE AWAY THE SLASH!

User-agent: *
Disallow:

More info:

Robotstxt.org documentation

BrownBailout.com

Why? First lets take a macro look at XML Sitemaps:

Search Engines agreed on a standard format for these sitemaps when they realized that, with many sites containing well beyond a million pages, the growing web was going to take a lot longer to crawl and index.  They proposed Sitemaps to allow for the following benefits:

1. More comprehensive indexing of large sites
2. Ability to understand whether a page has been updated without having to load the whole page
3. Ability to quickly understand how often a page is updated
4. Knowledge of the preferred URL to serve when choosing between two different duplicate pages

SEO Experts will see that I left “ability to understand importance of a page” off of the list.  Sitemaps protocol does include a tag that theoretically allows its creator to specify the “priority” of the page (from 0-1), but I have never seen an occaision where this arbitrary number took precedence over the search engine’s sophisticated ranking algorithm.

Now lets take a macro look at Sitemap Generators:

XML Sitemap generators are simple web crawlers that all follow a basic process:

1. User points crawler at their homepage URL, http://www.example.com for example
2. Crawler loads homepage HTML, looks for all links to other internal pages.
3. Crawler loads HTML from links it found, looks for more internal links.
4. Repeat…

The crawler ends up spitting out a nicely formatted XML sitemap that includes all the links it found in its 2 minute crawl of your site.

Now you see why Sitemap Generators are worthless: By design, they cannot provide any value that Search Engine’s sophisticated crawlers don’t provide already.

Who do you trust to do a smarter crawl of your site: Joe’s Ad-Supported XML Sitemap Generator, or the GoogleBot, featuring more than 10 years of relentless improvement and evolution?

So what is a webmaster to do? Here’s a nice Flowchart that will help you decide:

XML Sitemaps decision map

Note that there is NO path that ends in using an Online XML Sitemap Generator.

1. According to Google, Wikipedia.org gets around 28 million unique visitors per DAY
2. According to Hitwise, Google, and other Search Engines send Wikipedia 70% of its total traffic
3. Thus, Search Engines send Wikipedia approximately 18.9 million unique visitors each day!

Now the logic:

Two simple concepts make Wikipedia the site of choice for 19 million searches every day:

1. Search Engines place a lot of weight on “Anchor Text” or the text found linking to a page.
2. The rise of Blogging and Wikipedia Ubiquity

Part 1: Anchor Text

This one is pretty elementary, and begins with Sergei Brin and Lawrence Page’s idea that became Google. They decided to bring order to the web with a “link graph” that would rank pages by popularity.  One of the most useful effects of keeping track of all the links to a page is that one can also look at how other people describe that page.  For example, if a bunch of people link to ebaumsworld.com with the text “this hilarious site” (like this: I LOLed at this hilarious site), Google will decide: “Hey, we should probably show this site to people searching for ‘this hilarious site’.” And they do!

(This is where the Google Bomb comes from, a bunch of people coordinate an effort to link to a page with the exact same anchor text, such as “miserable failure” in the hopes that the page then ranks for that query on Google. Google now prevents this from happening by validating that the text in question also appears somewhere on the site.)

Part 2. The rise of Blogging

Bloggers, for better or worse, have been around since around 1999, but their numbers grew exponentially as blogging platforms such as Blogger and WordPress became easier and easier to use. Any good blogger will drop multiple links in every post they write.

A few of the most common reasons blogs link are:

1. Credit – another blog or website brought the topic to the blogger’s attention.
2. Definitive Source – If the blogger’s topic clearly has one authoritative source, www.pepsi.com when talking about Pepsi, for example.
3. Reference – If the blogger feels readers might need to know more about a topic, but is too lazy/doesn’t want to explain the topic in post.

This third reason is the one that gives Wikipedia 28 million visitors per day. Blog’s want to send visitors to an impartial accurate, and trustworthy explanation of the topic in question, so they look for the Wikipedia article, and link to it.

Conclusion:

The secret to Wikipedia’s search dominance is in the combination of the two concepts. When a blogger links to a Wikipedia article for reference, they simply highlight the term that needs to be referenced, and create the link. (IE: Then in 1863, came a turning point, the battle of gettysburg)

When linking to Wikipedia, the blogger doesn’t feel he needs to give any disclaimers or additional information about the link, because its leading to a clear, recognizable, and impartial source.

In contrast, if a blogger wanted to link to a non-wikipedia page, they might have to set up the link, describe it in more detail, or make a disclaimer first. (IE: Then in 1863, came a turning point, the Battle of Gettysburg. (See the Army’s interactive Gettysburg page here.)). It’s rare that someone would link to the army page wiith only “Battle of Gettysburg” because frankly there’s more to it then that.