alanthing

Linux and other things

Robots.txt Disallow All With Nginx

If you’re managing an environment similar to a production and want to keep bots from indexing traffic, it’s customary to add a robots.txt file at the root of your website to disallow all. Instead of creating a two-line plain text file, you can do this with only nginx:

1
2
3
4
location = /robots.txt {
  add_header  Content-Type  text/plain;
  return 200 "User-agent: *\nDisallow: /\n";
}

Add this into your configuration management as determined by environment, or add it by hand, and no longer worry if Google might start broadcasting your dev site to the world.

Comments