This tips is for webmaster who runs Wordpress, Jomla, or other related engine. Hmm what do you mean anita? everyone running that engine!, is there other engine besides those ?.. silly right.

What I want to tell you is how fast search engine such as google, yahoo, msn crawl your site without any granted permission from you as the site owner. You’ll shock if they spidering your secret directory such as your secure folder, your admin script etc and indexing in google search engine page list. Try searching at google with keyword /wp-login.php/ and you’ll find many unsecure wp login url ready to be hacked.

So how to overcome it ?, just tell your robot.txt to do this job automatically. Here what you can do :

Create file robot.txt, insert this line :

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-content/
Disallow: /wp-includes/
Disallow: /wp-login.php/

and upload it.

You can add other directories that you dont want to be published and crawled by search engine.