Optimizing Robots.txt file and Drupal SEO

Optimizing robots.txt file has an important role in the Drupal SEO. robotss.txt is a file that tell the search engine crawler whether a page should be crawled or not. The Drupal site has a default robotss.txt file in it's root folder. But as our drupal site develops, the robotss.txt file has to optimized to make our site Search Engine friendly. Optimizing robotss.txt file requires mainly when you use modules such as Views or Panels. This is because these modules creates duplicate content in your drupal site, which is harmful for SEO.


For example if you use a view with all content tagged under “Tag1”, then the contents comes under the “Tag1” display both in the View page and in the individual articles. Google bot and other search engine crawlers consider this as a duplication of data. So your search engine rank will surely go down. So the only thing we can do is to optimize robotss.txt file to hide the duplicated data from Search Engine crawlers.

Let you have two articles named as node/1 and node/2 which are tagged under “Tag1”. And also you have a view which contains all the articles tagged under “Tag1”. Now the content of node/1 articles displays both in .your-site.com/node/1” and “your-site.com/viewpage”. If you are not supposed to avoid the View, then you should optimize your robotss.txt file. Since view is more important to us than the articles, we have to block the crawler from accessing the individual articles. For this goto your root folder and add the following piece of code :


Disallow: /node/1
Disallow: /node/2

where node/1 and node/2 are the articles comes under the views. This code is valid if you turned on Clean URLs. Else add the following code for non – clean URLs.

Disallow: /?q=node/1
Disallow: /?q=node/2

Optimizing robots.txt file like this will avoid the duplication of data and Now your site is Search Engine Friendly. See also how to configure the Xml SiteMap for drupal SEO