What’s the Problem?

Do a quick Google search for Brown Bailout and take a look at the first few results.

Notice how BrownBailout.com's organic listing has no description.

It’s  easy to miss the first organic result, the link to the official webpage Brownbailout.com. The listing has only a basic two-word title, and no-description or cache links.  Not only that, but a search for site:brownbailout.com (to reveal all the pages within the site) turns up only that page, and nothing more!

Why did it happen?

The simplest SEO mistake in the book. Literally one character is costing BrownBailout.com thousands of visitors and advertising dollars. The culprit can be found here: http://www.brownbailout.com/robots.txt and looks like this:

User-agent: *
Disallow: /

This robots.txt is a little text file that sits on the server telling Search Engines “Stay Away!”  Typically this is placed on sites or sections of sites that have content that is expired, redundant or inferior. I am guessing someone from the PR or Advertising Agency that created BrownBailout.com left it up by mistake.  Sometimes misinformed developers or designers use the “User-agent: * Disallow: /” code as a way of soft-launching the site for a private group of users to see. This is very bad procedure for two reasons:

1. Not Secure: Anyone can type in the address and find the “private” site. Often times newly purchased domains come up on certain site’s radars automatically, creating links to the private site.
2. Easy to forget about: With the myriad other steps to worry about during a launch, a simple hidden txt file is very easy to overlook.

What are the effects?

Here is what the Google results page would look like if they fixed the robots.txt:

BrownBailout.com could own way more results page real estate if they fixed their SEO problem.

This would cause two major improvements:

1. Increase CTR on BrownBailout.com with more links and more real estate – that’s a no-brainer
2. Decrease CTR on negative articles and UPS paid ad – the extra sitelinks for BrownBailout.com attract a users attention away from the negative UPS ad, and push down the negative organic search result that is currently in third position.

How to fix it?

Come on dummy, take down the robots.txt file! Or even better, just TAKE AWAY THE SLASH!

User-agent: *
Disallow:

More info:

Robotstxt.org documentation

BrownBailout.com

Google’s policies and ranking algorithms are the unseen hand that shapes the web. Much in the same way that the US Gov’s subsidization of corn and tariff on foreign sugar has the effect of inundating American food with High-Fructose Corn Syrup, seemingly inconsequential decisions at Google ripple through the web in the form of widely perpetrated bad-design trends.

This article will break down the first of two trends, identify examples of sites that use this, analyze why it’s used, and give ideas on how to reverse it.

The Awkward Link

What is it?
The awkward link is a result of two concepts: 1. Google’s usage of anchor text in deciding rankings, and 2. Long-Tail user search trends that can be generalized into pattern form. See below for example:

Example of 'awkwardly-long' link spam perpetuated by Google's ranking algorithm

Notice how many times ‘restaurants’ is repeated? And then ‘cheap’ as well! Instead of thinking about the on-site experience, the designers of this site architected the links to match up with common local search patterns.

Why does it happen?

Heres how this link-puke usually comes about:

1. A company spots a search trend in a particular vertical: e.g.  ’[RESTAURANT/NIGHTLIFE/BAR/PUB] in [NEIGHBOURHOOD]‘
2. Ranking for one of these searches, ‘restaurants in east village’ for example, will bring in very little traffic on its own, so they have to go after thousands of these combinations.
3. They automate this process by gathering a big list of variations on ‘restaurant’ and a big list of neighborhoods, combining the two and printing them out over and over again.
4. When someone types in ‘cheap bars in south camden’, they’ve got a crappy ad-filled (see below) landing page ranking just for that term!

Who uses it?

Awkward link-puke is the meta-keywords field of 2010, everyone is going crazy with it.  Basically, if you’re not a recognizable brand, you have a lot of content and you want to get it ranked, you gotta use awkward links, here is a quick list:

Local Info Sites:

• Foodbuzz.com
• ViewLondon.co.uk
• SuperPages.com
• InsiderPages.com

Celebrity:

• FanPix.net – Don’t ask me how I found this

Jobs:

• Yahoo HotJobs – YAHOO?
• Indeed.com – Not as bad as others

Also susceptible to link puke: Real Estate, Online Dating, and more drop a link in the comments if you have any good examples.

Why is it bad?

It’s bad for user experience.  It demonstrates the fundamental mismatch in how a user navigates a site, and how a Search Engine indexes a site:

A user has the advantage of context: they can start on the homepage, click ‘Manhattan’ then ‘Restaurants’, then ‘Cheap’, then ‘Chinese’ and they know they are in ‘Cheap Chinese Restaurants in Manhattan’.

A Search Engine is constrained to relatively simple rules, their bots prefer to see a link that says: ‘Cheap Chinese Restaurants in Manhattan’ because it is more specific. They don’t care that it’s stuffed into a bank of 100 similarly long links.

How to reverse the trend

Smarter content recognition, and better use of ‘user satisfaction’ metrics.

Search Engines crawlers already claim to do a certain amount of smart content recognition. Google decides that certain pages and links are important enough to highlight in sitelinks, Yahoo has said that they would discount a link down in the footer, and Microsoft has published research that attempts to isolate the unique content on a page. But they need to kick things into overdrive.  Crawlers need to get beyond the page, and think about the site as a whole like a user would.  In the meantime though, why not just tweak the PageRank algorithm to falloff quickly after 500 words of anchor text.  Right now outgoing PageRank is calculated based on the number of links, why not factor in the total amount of anchor text on the page as well?

When I hit one of the sites listed above, stuffed with link-puke and sneaky ads, I immediately hit my back button and click another result. Google is technically able to record this ‘dissatisfaction’ action, but my habits may not reflect those of the masses. Either way, Google doesn’t seem to be using this information well, given the amount of crap in search results.

Conclusion

Until Search Engine’s make changes to their ranking algorithm, usage of awkward links will continue to grow. Actually, the business benefit of this technique is only growing right now, given the trend towards longer, more detailed searches.

Recently Google added the functionality to include images in your XML Sitemaps.

This isn’t particularly world-changing news, but it may mean that complex image URLs can finally be indexed and found by Google’s Image bot.

Up until now, image URL’s that didn’t end in .jpg, .png, .gif, .bmp, etc… couldn’t be found and indexed in Google’s Image Search. I have set up a Complex Image URL Google Test to try and determine whether the new XML Sitemap with Image data allows webmasters to get complex image URLs found and ranked in Google Image Search.

Did you see it?

Let’s take a look at the User’s View vs Search Engine’s View side-by-side:

(see the big blue text: “Google Phone”)

I must admit it’s not the most heinous of SEO violations, but I would call it a guidelines violation nevertheless: Google uses a sneaky CSS technique to show Search Engines an <H1> tag that says “Google Phone” in place of what users see as the “Nexus One” logo. In its Webmaster Guidelines, Google says:

Hiding text or links in your content can cause your site to be perceived as untrustworthy since it presents information to search engines differently than to visitors. Text (such as excessive keywords) can be hidden in several ways, including:

• Using CSS to hide text

To be fair, I don’t think Google is really gaining anything from this, unless they are using it to get around the anti-Googlebombing measure that was implemented a while ago. But that’s just dangerous speculation. I am guessing the real story is that not everyone at Google is an SEO geek, and some poor wireframe monkey did it because that’s what he/she has always done.

What exactly is Google doing wrong?

Google is using a ghetto version of CSS image replacement to tell Search Engines: “Hey I should rank for ‘Google Phone’ cuz that’s what people search for and it’s in my H1 tag” while telling users: “Google Phone is so 2007, call this new gadget the Nexus One.”

Basically, somewhere deep within a CSS file (here), there is code that sets a text-indent:-5000px style on the H1 tag text. This effectively places the text so far off screen that no one but this guy would possibly ever see it.

What do you think? Is this just another case of SEO “playa hatin”? Is it not even a violation of the Webmaster Guidelines? Should i be scared of the Nexus?

Why? First lets take a macro look at XML Sitemaps:

Search Engines agreed on a standard format for these sitemaps when they realized that, with many sites containing well beyond a million pages, the growing web was going to take a lot longer to crawl and index.  They proposed Sitemaps to allow for the following benefits:

1. More comprehensive indexing of large sites
2. Ability to understand whether a page has been updated without having to load the whole page
3. Ability to quickly understand how often a page is updated
4. Knowledge of the preferred URL to serve when choosing between two different duplicate pages

SEO Experts will see that I left “ability to understand importance of a page” off of the list.  Sitemaps protocol does include a tag that theoretically allows its creator to specify the “priority” of the page (from 0-1), but I have never seen an occaision where this arbitrary number took precedence over the search engine’s sophisticated ranking algorithm.

Now lets take a macro look at Sitemap Generators:

XML Sitemap generators are simple web crawlers that all follow a basic process:

1. User points crawler at their homepage URL, http://www.example.com for example
2. Crawler loads homepage HTML, looks for all links to other internal pages.
3. Crawler loads HTML from links it found, looks for more internal links.
4. Repeat…

The crawler ends up spitting out a nicely formatted XML sitemap that includes all the links it found in its 2 minute crawl of your site.

Now you see why Sitemap Generators are worthless: By design, they cannot provide any value that Search Engine’s sophisticated crawlers don’t provide already.

Who do you trust to do a smarter crawl of your site: Joe’s Ad-Supported XML Sitemap Generator, or the GoogleBot, featuring more than 10 years of relentless improvement and evolution?

So what is a webmaster to do? Here’s a nice Flowchart that will help you decide:

XML Sitemaps decision map

Note that there is NO path that ends in using an Online XML Sitemap Generator.

1. According to Google, Wikipedia.org gets around 28 million unique visitors per DAY
2. According to Hitwise, Google, and other Search Engines send Wikipedia 70% of its total traffic
3. Thus, Search Engines send Wikipedia approximately 18.9 million unique visitors each day!

Now the logic:

Two simple concepts make Wikipedia the site of choice for 19 million searches every day:

1. Search Engines place a lot of weight on “Anchor Text” or the text found linking to a page.
2. The rise of Blogging and Wikipedia Ubiquity

Part 1: Anchor Text

This one is pretty elementary, and begins with Sergei Brin and Lawrence Page’s idea that became Google. They decided to bring order to the web with a “link graph” that would rank pages by popularity.  One of the most useful effects of keeping track of all the links to a page is that one can also look at how other people describe that page.  For example, if a bunch of people link to ebaumsworld.com with the text “this hilarious site” (like this: I LOLed at this hilarious site), Google will decide: “Hey, we should probably show this site to people searching for ‘this hilarious site’.” And they do!

(This is where the Google Bomb comes from, a bunch of people coordinate an effort to link to a page with the exact same anchor text, such as “miserable failure” in the hopes that the page then ranks for that query on Google. Google now prevents this from happening by validating that the text in question also appears somewhere on the site.)

Part 2. The rise of Blogging

Bloggers, for better or worse, have been around since around 1999, but their numbers grew exponentially as blogging platforms such as Blogger and WordPress became easier and easier to use. Any good blogger will drop multiple links in every post they write.

A few of the most common reasons blogs link are:

1. Credit – another blog or website brought the topic to the blogger’s attention.
2. Definitive Source – If the blogger’s topic clearly has one authoritative source, www.pepsi.com when talking about Pepsi, for example.
3. Reference – If the blogger feels readers might need to know more about a topic, but is too lazy/doesn’t want to explain the topic in post.

This third reason is the one that gives Wikipedia 28 million visitors per day. Blog’s want to send visitors to an impartial accurate, and trustworthy explanation of the topic in question, so they look for the Wikipedia article, and link to it.

Conclusion:

The secret to Wikipedia’s search dominance is in the combination of the two concepts. When a blogger links to a Wikipedia article for reference, they simply highlight the term that needs to be referenced, and create the link. (IE: Then in 1863, came a turning point, the battle of gettysburg)

When linking to Wikipedia, the blogger doesn’t feel he needs to give any disclaimers or additional information about the link, because its leading to a clear, recognizable, and impartial source.

In contrast, if a blogger wanted to link to a non-wikipedia page, they might have to set up the link, describe it in more detail, or make a disclaimer first. (IE: Then in 1863, came a turning point, the Battle of Gettysburg. (See the Army’s interactive Gettysburg page here.)). It’s rare that someone would link to the army page wiith only “Battle of Gettysburg” because frankly there’s more to it then that.

1. Learning through examples.
2. Exposing unethical SEO to bring more attention to it, and stop it.
3. Fostering discussion of SEO techniques (cause we all know there’s not nearly enough SEO discussion)
4. Determining what works, what doesn’t.

“You better not publish my secret SEO Techniques!” – If you find yourself saying this, and you’re not making money by doing things you wouldn’t tell your mum, then post a comment here and we can discuss.