Search Crawlers, Controlling The Behavior of Spiders in Your Page

By On May 23, 2011 • Search Engine Optimization Comments (0)

nofollow-noindex-rel-seotekyThe topic was on, when someone asked “talaga bang me rel=dofollow?” Do rel=dofollow exist? Technically no, theres no such thing as rel=dofollow. If you write your tag, the rel property is already on follow by default, else if you use rel=nofollow then the link is on no follow link attribute.

This topic existed when pagerank sculpting was on the fold in search engine optimization. Manipulating the flow of the pagerank juice in your page by using nofollow rel attribute. Rel=nofollow was introduced 2005 and supported by different search engines. Nowadays nofollow rel attribute is used to reduce spam and most common on blogging platform such as wordpress. To further this let me define the different ways to implement nofollow and noindex in your HTML.

<a href=”” rel=”nofollow”>

This tell search engine not to follow the specific link mentioned above.

Social circle existed and nofollow rel attribute now affects Social Graphs via Social Graphs API and used “me” to indicate that its your social profile available in the web.

<a href=”” rel=”nofollow me”>

Controlling the crawlers through meta details

The following codes affects the whole page since it is required to be used inside <head></head> HTML code. The term index means put the page on a search result page – Google, Yahoo, Bing etc.

<meta name=”robots” content=”index,follow”>

This tell search engines to index the page and follow the links on that page. A typical setting on a hand coded HTML website.

<meta name=”robots” content=”noindex,follow”>

This tell search engines not to index the page but follow all the links on the page. This one exist sometimes on forum private messages.

<meta name=”robots” content=”index,nofollow”>

This tell search engines to index the page in search result pages but not to follow the links on the page.

<meta name=”robots” content=”noindex,nofollow”>

This tell search engines not to index the page and not to follow the links on the particular page.

Based from my read, Matt cutts said that using nofollow rel attribute on robots.txt doesn’t give you an assurance that your page will not be indexed or not followed and can still get pagerank.

Now, robots.txt says you are not allowed to crawl a page, and Google therefore does not crawl pages that are forbidden in robots.txt. However, they can accrue PageRank, and they can be returned in our search results.

More about Eric Enge and Matt Cutts – Oct 8, 2007

Now on Matt Cutts take on PageRank sculpting

I wouldn’t recommend it, because it isn’t the most effective way to utilize your PageRank. In general, I would let PageRank flow freely within your site. The notion of “PageRank sculpting” has always been a second- or third-order recommendation for us. I would recommend the first-order things to pay attention to are 1) making great content that will attract links in the first place, and 2) choosing a site architecture that makes your site usable/crawlable for humans and search engines alike.

Matt Cutts on the blog



Jayson Bagio is the lead strategist of SEOteky. A digital marketing team from Manila, Ph who specializes in content marketing and link building. Get in touch with him using the contact form on the side or by visiting the about page. You can also follow him on twitter & google+.

join conversation

No related posts.

Leave a Reply

Your email address will not be published. Required fields are marked *


You may use these HTML tags and attributes: <a href="" title="" rel=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

CommentLuv badge

« »