Control Search Engine Robots

Posted on May 5, 2008
Filed Under All |

Controlling how search engine robots index and access your blog is totally up to you as the blog owner. Search engine robots are very obedient, and you can tell them what pages to crawl what pages and/directories to leave alone. Search engine robots understand exactly what we are telling them because we do so by using the robots.txt Robots Exclusion Standard Protocol language.

Dan Crow over at Google’s official site wrote a great article, “Controlling how search engines access and index your website”

Visit DailyBlogTip’s collection of Robot.txt files for a wide range of big name blogs and websites. Also, he shows you how to make a robots.txt file.

Share and Enjoy: These icons link to social bookmarking sites where readers can share and discover new web pages.