Basic robots.txt for WordPress

1. Never allow indexing of the cgi-bin


User-agent: *
Disallow: /cgi-bin

2. Never allow indexing WordPress folders


Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes

If you want to allow the  wp-content/uploads folder


Allow: /wp-content/uploads

3. Block feeds, this will make the robots crawling on-site content, gives better ranks in the SEARCH ENGINE RESULT PAGE


Disallow: /feed
Disallow: */feed

4. Never allow indexing trackback


Disallow: /trackback
Disallow: */trackback

5. Make robots think comments as part of the on site content, not in comment feed


Disallow: /comments
Disallow: */comments

6. Never allow indexing XML-RPC file to avoid hole in security


Disallow: /xmlrpc.pp

7. Disallow file types for better security

Disallow: /*.php$
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Disallow: /*.gz$
Disallow: /*.wmv$
Disallow: /*.cgi$
Disallow: /*.xhtml$
Disallow: /*.xlsx$
Disallow: /*.doc$
Disallow: /*.pdf$
Disallow: /*.zip$
Disallow: /*?*
Disallow: /*?

8. Use Google Webmaster Tools to make sure robots.txt configuration doesn’t give problem when Googlebot indexing your WordPress site. Google Webmaster Tools could identify problem in robots.txt and let you know about the problem.

%d bloggers like this: