Thursday, November 21, 2013

Restrict Google Search Crawl

STEPS or CODE GENERATOR to Restrict Search Engines

1.  New File named:  robots.txt

2.  Add this text:

# robots.txt generated at http://www.mcanerin.com
User-agent: *
Disallow: /

3.  Copy to Web Server Root (in /var/www/ for apache on linux)


NOTE: This will restrict all search engines from crawling and indexing your web server so that it doesn't show up on Google or Bing results.  #3 is specific to Linux Ubuntu with Apache Web Server installed.

**GOOGLE ANALYTICS WILL STILL WORK**  
Blocking search engines with robots.txt does not affect Google Analytics.  As long as the analytics UA tracking code script is still in the HTML file, Google Analytics will track visits.