Prevent search engines from crawling and indexing your website
All search engines have automatic web crawlers that are constantly browsing the internet in order to keep their search results up to date. Unless you specifically configure your website to disallow them, they will eventually find you and list you in their search results.
To disallow these crawlers(robots) from crawling your website, just create a file called “robots.txt” with the following data.
User-agent: *
Disallow: /
Upload the “robots.txt” file to the root of your website and it will prevent the crawlers from crawling your website.
You can also use meta tags to prevent the robots from crawling a specific page. To do so, place the following code within the <HEAD></HEAD> tags of your html.
<meta name=”ROBOTS” content=”NOINDEX, NOFOLLOW”>
The NOINDEX keyword is telling the robots not to index the current page in their search results. The NOFOLLOW keyword is telling the robots not to follow any links on the current page.