Robots.txt Generator


Default - All Robots are:  
    
Crawl-Delay:
    
Sitemap: (leave blank if you don't have) 
     
Search Robots: Google
  Google Image
  Google Mobile
  MSN Search
  Yahoo
  Yahoo MM
  Yahoo Blogs
  Ask/Teoma
  GigaBlast
  DMOZ Checker
  Nutch
  Alexa/Wayback
  Baidu
  Naver
  MSN PicSearch
   
Restricted Directories: The path is relative to root and must contain a trailing slash "/"
 
 
 
 
 
 
   



Now, Create 'robots.txt' file at your root directory. Copy above text and paste into the text file.


About Robots.txt Generator

Robots.Txt A Guide for Crawlers - Use Google Robots Txt Generator

The robots.txt file is used by websites to tell bots which parts of their site need to be crawled and indexed. This standard is also known as the robots exclusion protocol. You can use this file to specify which areas of your site you don't want to be processed by crawlers, such as areas containing duplicate content or under development. However, not all bots follow this standard, so there's a chance that the bot will begin crawling your site from the areas you don't want to be indexed.