STEPS or CODE GENERATOR to Restrict Search Engines
1. New File named: robots.txt
2. Add this text:
# robots.txt generated at http://www.mcanerin.com
User-agent: *
Disallow: /
3. Copy to Web Server Root (in /var/www/ for apache on linux)
1. New File named: robots.txt
2. Add this text:
# robots.txt generated at http://www.mcanerin.com
User-agent: *
Disallow: /
3. Copy to Web Server Root (in /var/www/ for apache on linux)
NOTE: This will restrict all search engines from crawling and indexing your web server so that it doesn't show up on Google or Bing results. #3 is specific to Linux Ubuntu with Apache Web Server installed.
**GOOGLE ANALYTICS WILL STILL WORK**
Blocking search engines with robots.txt does not affect Google Analytics. As long as the analytics UA tracking code script is still in the HTML file, Google Analytics will track visits.