Thursday, November 21, 2013

Restrict Google Search Crawl

STEPS or CODE GENERATOR to Restrict Search Engines

1.  New File named:  robots.txt

2.  Add this text:

# robots.txt generated at
User-agent: *
Disallow: /

3.  Copy to Web Server Root (in /var/www/ for apache on linux)

NOTE: This will restrict all search engines from crawling and indexing your web server so that it doesn't show up on Google or Bing results.  #3 is specific to Linux Ubuntu with Apache Web Server installed.

Blocking search engines with robots.txt does not affect Google Analytics.  As long as the analytics UA tracking code script is still in the HTML file, Google Analytics will track visits.