1. Never allow indexing of the cgi-bin<\/p>\n
\r\n\r\nUser-agent: *\r\nDisallow: \/cgi-bin\r\n\r\n<\/pre>\n2. Never allow indexing WordPress folders<\/p>\n
\r\n\r\nDisallow: \/wp-admin\r\nDisallow: \/wp-includes\r\nDisallow: \/wp-content\/plugins\r\nDisallow: \/wp-content\/cache\r\nDisallow: \/wp-content\/themes\r\n\r\n<\/pre>\nIf you want to allow the \u00a0wp-content\/uploads folder<\/p>\n
\r\n\r\nAllow: \/wp-content\/uploads\r\n\r\n<\/pre>\n3. Block feeds, this will make the robots crawling on-site content, gives better ranks in the SEARCH ENGINE RESULT PAGE<\/p>\n
\r\n\r\nDisallow: \/feed\r\nDisallow: *\/feed\r\n\r\n<\/pre>\n4. Never allow indexing trackback<\/p>\n
\r\n\r\nDisallow: \/trackback\r\nDisallow: *\/trackback\r\n\r\n<\/pre>\n5. Make robots think comments as part of the on site content, not in comment feed<\/p>\n
\r\n\r\nDisallow: \/comments\r\nDisallow: *\/comments\r\n\r\n<\/pre>\n6. Never allow indexing XML-RPC file to avoid hole in security<\/p>\n
\r\n\r\nDisallow: \/xmlrpc.pp\r\n\r\n<\/pre>\n7. Disallow file types for better security<\/p>\n
\r\nDisallow: \/*.php$\r\nDisallow: \/*.js$\r\nDisallow: \/*.inc$\r\nDisallow: \/*.css$\r\nDisallow: \/*.gz$\r\nDisallow: \/*.wmv$\r\nDisallow: \/*.cgi$\r\nDisallow: \/*.xhtml$\r\nDisallow: \/*.xlsx$\r\nDisallow: \/*.doc$\r\nDisallow: \/*.pdf$\r\nDisallow: \/*.zip$\r\nDisallow: \/*?*\r\nDisallow: \/*?\r\n<\/pre>\n8. Use Google Webmaster Tools to make sure robots.txt configuration doesn’t give problem when Googlebot indexing your WordPress site. Google Webmaster Tools could identify problem in robots.txt and let you know about the problem.<\/p>\n","protected":false},"excerpt":{"rendered":"
1. Never allow indexing of the cgi-bin User-agent: * Disallow: \/cgi-bin 2. Never allow indexing WordPress folders Disallow: \/wp-admin Disallow: \/wp-includes Disallow: \/wp-content\/plugins Disallow: \/wp-content\/cache Disallow: \/wp-content\/themes If you want to allow the \u00a0wp-content\/uploads folder Allow: \/wp-content\/uploads 3. Block feeds, this will make the robots crawling on-site content, gives better ranks in the SEARCH ENGINE […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[5],"tags":[57,78,79,6],"class_list":["post-183","post","type-post","status-publish","format-standard","hentry","category-wordpress","tag-google","tag-googlebot","tag-robots-txt","tag-wordpress-2"],"aioseo_notices":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p1pvi1-2X","jetpack-related-posts":[{"id":111,"url":"https:\/\/pixert.com\/blog\/how-to-disable-directory-browsing\/","url_meta":{"origin":183,"position":0},"title":"How to disable directory browsing?","author":"Pixel Insert \/ Pixert","date":"June 3, 2011","format":false,"excerpt":"One of the must do on setting a secure Apache webserver is to disable directory browsing. This could prevent the server from showing a listing of the existing files if there's no index in one folder. Disable directory browsing via .htaccess Add in the .htaccess file the following [bash] Options\u2026","rel":"","context":"In "CentOS"","block_context":{"text":"CentOS","link":"https:\/\/pixert.com\/blog\/category\/centos\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/pixert.com\/blog\/wp-content\/uploads\/2011\/03\/feather-small.gif?resize=350%2C200","width":350,"height":200},"classes":[]},{"id":195,"url":"https:\/\/pixert.com\/blog\/redirect-certain-pages-from-http-to-https-with-htaccess\/","url_meta":{"origin":183,"position":1},"title":"Redirect certain pages from http to https with .htaccess","author":"Pixel Insert \/ Pixert","date":"February 10, 2012","format":false,"excerpt":"The following trick is useful for E-Commerce sites. \u00a0We only need few pages redirected to https to secure transaction This will automatically rewrite the url to https as long as the page is not already an https link. This conditional keeps it from causing a loop. It's very handy to\u2026","rel":"","context":"In "WordPress"","block_context":{"text":"WordPress","link":"https:\/\/pixert.com\/blog\/category\/wordpress\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/pixert.com\/blog\/wp-content\/uploads\/2011\/03\/wordpresslogo-300x68.png?resize=350%2C200","width":350,"height":200},"classes":[]},{"id":117,"url":"https:\/\/pixert.com\/blog\/hide-the-apache-web-server-version-number\/","url_meta":{"origin":183,"position":2},"title":"Hide the Apache Web Server Version number","author":"Pixel Insert \/ Pixert","date":"June 6, 2011","format":false,"excerpt":"There are two config directives that controls Apache version. The\u00a0ServerSignature directive adds a line containing the Apache HTTP Server server version and the ServerName to any server-generated documents, such as error messages sent back to clients. The\u00a0ServerTokens directive controls whether Server response header field which is sent back to clients\u2026","rel":"","context":"In "CentOS"","block_context":{"text":"CentOS","link":"https:\/\/pixert.com\/blog\/category\/centos\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/pixert.com\/blog\/wp-content\/uploads\/2011\/03\/feather-small.gif?resize=350%2C200","width":350,"height":200},"classes":[]},{"id":170,"url":"https:\/\/pixert.com\/blog\/repair-wordpress-database\/","url_meta":{"origin":183,"position":3},"title":"Repair WordPress Database","author":"Pixel Insert \/ Pixert","date":"December 10, 2011","format":false,"excerpt":"When a WordPress table crash, one symptom might be that all of your posts seem to have disappeared, or perhaps all of your drafts. You may even get a message in phpMyAdmin telling you that one of your tables haves crashed. No need to worry. Repairing tables is very easy.\u2026","rel":"","context":"In "CentOS"","block_context":{"text":"CentOS","link":"https:\/\/pixert.com\/blog\/category\/centos\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/pixert.com\/blog\/wp-content\/uploads\/2011\/12\/phpmyadminrepair.jpg?fit=1064%2C640&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/pixert.com\/blog\/wp-content\/uploads\/2011\/12\/phpmyadminrepair.jpg?fit=1064%2C640&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/pixert.com\/blog\/wp-content\/uploads\/2011\/12\/phpmyadminrepair.jpg?fit=1064%2C640&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/pixert.com\/blog\/wp-content\/uploads\/2011\/12\/phpmyadminrepair.jpg?fit=1064%2C640&ssl=1&resize=700%2C400 2x, https:\/\/i0.wp.com\/pixert.com\/blog\/wp-content\/uploads\/2011\/12\/phpmyadminrepair.jpg?fit=1064%2C640&ssl=1&resize=1050%2C600 3x"},"classes":[]},{"id":82,"url":"https:\/\/pixert.com\/blog\/git-lesson-branch\/","url_meta":{"origin":183,"position":4},"title":"Git lesson : Branch","author":"Pixel Insert \/ Pixert","date":"April 12, 2011","format":false,"excerpt":"What is GIT? Git is... Git is a free & open source, distributed version control system designed to handle everything from small to very large projects with speed and efficiency. Every Git clone is a full-fledged repository with complete history and full revision tracking capabilities, not dependent on network access\u2026","rel":"","context":"In "Mac OS"","block_context":{"text":"Mac OS","link":"https:\/\/pixert.com\/blog\/category\/mac-os\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/pixert.com\/blog\/wp-content\/uploads\/2011\/04\/githeader.png?resize=350%2C200","width":350,"height":200},"classes":[]},{"id":63,"url":"https:\/\/pixert.com\/blog\/how-to-find-apache-version-on-linux\/","url_meta":{"origin":183,"position":5},"title":"How to find Apache Version on Linux?","author":"Pixel Insert \/ Pixert","date":"March 24, 2011","format":false,"excerpt":"Run the following to see Apache version running on your Linux system [bash]httpd -v[\/bash] or [bash]\/usr\/local\/apache\/bin\/httpd -v[\/bash]","rel":"","context":"In "CentOS"","block_context":{"text":"CentOS","link":"https:\/\/pixert.com\/blog\/category\/centos\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/pixert.com\/blog\/wp-content\/uploads\/2011\/03\/feather-small.gif?resize=350%2C200","width":350,"height":200},"classes":[]}],"jetpack_sharing_enabled":true,"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/pixert.com\/blog\/wp-json\/wp\/v2\/posts\/183","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/pixert.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/pixert.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/pixert.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/pixert.com\/blog\/wp-json\/wp\/v2\/comments?post=183"}],"version-history":[{"count":0,"href":"https:\/\/pixert.com\/blog\/wp-json\/wp\/v2\/posts\/183\/revisions"}],"wp:attachment":[{"href":"https:\/\/pixert.com\/blog\/wp-json\/wp\/v2\/media?parent=183"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/pixert.com\/blog\/wp-json\/wp\/v2\/categories?post=183"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/pixert.com\/blog\/wp-json\/wp\/v2\/tags?post=183"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}