diff options
| author | Aldo Cortesi <aldo@corte.si> | 2018-03-05 10:57:50 +1300 |
|---|---|---|
| committer | Aldo Cortesi <aldo@corte.si> | 2018-03-05 10:57:50 +1300 |
| commit | ee6937f948f96d74d413a24f89c2317ab582d0c8 (patch) | |
| tree | 4454b57a641a7508b73d440bf2c41d6fbc5f04f0 /docs/bucketassets/robots.txt | |
| parent | a68c7ffb27278c4e2aea4d2d8a87ee15a86bcd88 (diff) | |
| download | mitmproxy-ee6937f948f96d74d413a24f89c2317ab582d0c8.tar.gz mitmproxy-ee6937f948f96d74d413a24f89c2317ab582d0c8.tar.bz2 mitmproxy-ee6937f948f96d74d413a24f89c2317ab582d0c8.zip | |
docs: add robots.txt and error.html
- robots.txt prevents crawling of everything but /stable
- error.html has a simple meta refresh to the latest stable docs. This will
also help crawlers find the right path.
Diffstat (limited to 'docs/bucketassets/robots.txt')
| -rw-r--r-- | docs/bucketassets/robots.txt | 3 |
1 files changed, 3 insertions, 0 deletions
diff --git a/docs/bucketassets/robots.txt b/docs/bucketassets/robots.txt new file mode 100644 index 00000000..da4c47d6 --- /dev/null +++ b/docs/bucketassets/robots.txt @@ -0,0 +1,3 @@ +User-agent: * +Disallow: /archive/ +Disallow: /master/
\ No newline at end of file |