Your free guide to search engine optimization.
Most search engine spiders (Googlebot, Yahoo! Slurp, MSNBot, etc.) only visit the places on a website where they have permission to crawl. When spiders access a site, they check a special file in the root directory of the Web server called "robots.txt." This file implements the Robots Exclusion Protocol, which allows websites to declare the parts of a site that are off-limits to specific "crawlers" or even all spiders. Robots.txt is essentially a text based configuration file with the ability to specify which files or directories on the website should be crawled, and it can specify these options by individual spider if desired. Site owners can disallow access to .cgi, private, temporary directories, and other areas with pages they do not want accessed or indexed.
The robots.txt file must be located on the Web server's root directory. For instance, if the site is
located at seo.babyloncreations.com, a file needs to be created and located here:
seo.babyloncreations.com/robots.txt.
