Tag: Googlebot

  • Basic robots.txt for WordPress

    1. Never allow indexing of the cgi-bin 2. Never allow indexing WordPress folders If you want to allow the  wp-content/uploads folder 3. Block feeds, this will make the robots crawling on-site content, gives better ranks in the SEARCH ENGINE RESULT PAGE 4. Never allow indexing trackback 5. Make robots think comments as part of the…