1. Never allow indexing of the cgi-bin
User-agent: * Disallow: /cgi-bin
2. Never allow indexing WordPress folders
Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /wp-content/cache Disallow: /wp-content/themes
If you want to allow the wp-content/uploads folder
Allow: /wp-content/uploads
3. Block feeds, this will make the robots crawling on-site content, gives better ranks in the SEARCH ENGINE RESULT PAGE
Disallow: /feed Disallow: */feed
4. Never allow indexing trackback
Disallow: /trackback Disallow: */trackback
5. Make robots think comments as part of the on site content, not in comment feed
Disallow: /comments Disallow: */comments
6. Never allow indexing XML-RPC file to avoid hole in security
Disallow: /xmlrpc.pp
7. Disallow file types for better security
Disallow: /*.php$ Disallow: /*.js$ Disallow: /*.inc$ Disallow: /*.css$ Disallow: /*.gz$ Disallow: /*.wmv$ Disallow: /*.cgi$ Disallow: /*.xhtml$ Disallow: /*.xlsx$ Disallow: /*.doc$ Disallow: /*.pdf$ Disallow: /*.zip$ Disallow: /*?* Disallow: /*?
8. Use Google Webmaster Tools to make sure robots.txt configuration doesn’t give problem when Googlebot indexing your WordPress site. Google Webmaster Tools could identify problem in robots.txt and let you know about the problem.