All search engines have automatic web crawlers that are constantly browsing the internet in order to keep their search results up to date. Unless you specifically configure your website to disallow them, they will eventually find you and list you in their search results. To disallow these crawlers(robots) from crawling your website, just create a [...]
Archive | Search engines RSS feed for this section
Prevent search engines from crawling and indexing your website
by admin on 29. Sep, 2008 in Html, Search engines, Web
-
Understanding Linux File Permissions
03. Oct, 2008
-
How to setup a redundant NFS server with DRBD and Heartbeat in CentOS 5
18. Jan, 2009
-
Automatically Add WWW to All Your URLs
04. Oct, 2008
-
How to Install ThinkorSwim in Linux
11. Oct, 2008
-
How to setup a redundant NFS server with DRBD and Heartbeat in CentOS 5
18. Jan, 2009
-
Error: Too many open files
28. Oct, 2008
-
How to Reset the MySQL Root Password
14. Oct, 2008
-
How to Install ThinkorSwim in Linux
11. Oct, 2008