diff options
| author | James Mills <prologic@shortcircuit.net.au> | 2018-04-03 14:32:43 -0700 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2018-04-03 14:32:43 -0700 |
| commit | fe18a79dd2f7115b30d440a025a7f7f72486dd09 (patch) | |
| tree | a2a5c9f24f90837013c6407077f687bf47de4934 /robots.txt | |
| parent | Removed build artifacts accidently committed to dist/ (diff) | |
| download | gopherproxy-fe18a79dd2f7115b30d440a025a7f7f72486dd09.tar.gz gopherproxy-fe18a79dd2f7115b30d440a025a7f7f72486dd09.tar.bz2 gopherproxy-fe18a79dd2f7115b30d440a025a7f7f72486dd09.zip | |
Add support for robots.txt user agent control from web crawlers (#10)
Diffstat (limited to 'robots.txt')
| -rw-r--r-- | robots.txt | 2 |
1 files changed, 2 insertions, 0 deletions
diff --git a/robots.txt b/robots.txt new file mode 100644 index 0000000..1f53798 --- /dev/null +++ b/robots.txt | |||
| @@ -0,0 +1,2 @@ | |||
| 1 | User-agent: * | ||
| 2 | Disallow: / | ||