Robots.txt for WordPress
For those of you who do not know what a robots.txt file does, it basically tells the robots which pages on your site they can and can not crawl. This is important because there are certain things you do not want to show up in the index, like if a folder contains duplicate content, or it's a private membership site then you don't want google giving away information for free that you are charging people for. Here is the current edition of the robots.txt file I use for all of my sites, both to give the search engine robots access to my sitemap and also to block them from directories I don't want them having access to:
Sitemap: http://www.jamiefaidley.com/sitemap.xml
User-agent: *
Disallow: /cgi-bin
Disallow: /wp-includes
Disallow: /wp-content/cache
Disallow: /wp-content/plugins
Disallow: /wp-content/themes
Disallow: */trackback
Disallow: */comments
Disallow: /*?*
Disallow: /*?
Sitemap: http://www.jamiefaidley.com/sitemap.xml
User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /images/
Disallow: /wp-includes
Disallow: /wp-content/cache
Disallow: /wp-content/plugins
Disallow: /wp-content/themes
Disallow: */trackback
Disallow: */comments
Disallow: /*?*
Disallow: /*?
If you liked this article, you may also be interested in: