Basic robots.txt for WordPress

1. Never allow indexing of the cgi-bin


User-agent: *
Disallow: /cgi-bin

2. Never allow indexing WordPress folders


Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes

If you want to allow the  wp-content/uploads folder


Allow: /wp-content/uploads

3. Block feeds, this will make the robots crawling on-site content, gives better ranks in the SEARCH ENGINE RESULT PAGE


Disallow: /feed
Disallow: */feed

4. Never allow indexing trackback


Disallow: /trackback
Disallow: */trackback

5. Make robots think comments as part of the on site content, not in comment feed


Disallow: /comments
Disallow: */comments

6. Never allow indexing XML-RPC file to avoid hole in security


Disallow: /xmlrpc.pp

7. Disallow file types for better security

Disallow: /*.php$
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Disallow: /*.gz$
Disallow: /*.wmv$
Disallow: /*.cgi$
Disallow: /*.xhtml$
Disallow: /*.xlsx$
Disallow: /*.doc$
Disallow: /*.pdf$
Disallow: /*.zip$
Disallow: /*?*
Disallow: /*?

8. Use Google Webmaster Tools to make sure robots.txt configuration doesn’t give problem when Googlebot indexing your WordPress site. Google Webmaster Tools could identify problem in robots.txt and let you know about the problem.

Google Safe Browsing Diagnostic Tool helps a site owner detect malware

Site Owner could do a quick malware check on their siteor any other website using Google Safe Browsing Diagnostic tool

 

 

 

 

 

 

 

 

Visit the following URL and change the domain name in the end

http://www.google.com/safebrowsing/diagnostic?site=domainname.com

It will tell you what type of data is stored about a compromised site anyway. Google’s Diagnostics answers four questions about a compromised site :

  • What is the current listing status for [the site in question]?
  • What happened when Google visited this site?
  • Has this site acted as an intermediary resulting in further distribution of malware?
  • Has this site hosted malware?