Tag: Googlebot
Basic robots.txt for WordPress
1. Never allow indexing of the cgi-bin 2. Never allow indexing WordPress folders If you want to allow the wp-content/uploads folder 3. Block feeds, this will make the robots crawling on-site content, gives better ranks in the SEARCH ENGINE RESULT PAGE 4. Never allow indexing trackback 5. Make robots think comments as part of the…