Merge pull request #4801 from opensafely-core/slow-down-bots

Increase bot crawl delay
opensafely-core · Jan 13, 2025 · 1fbe102 · 1fbe102
2 parents d4a16f1 + d1d0d82
commit 1fbe102
Showing 1 changed file with 1 addition and 1 deletion.
diff --git a/static/robots.txt b/static/robots.txt
@@ -1,5 +1,5 @@
 User-agent: *
-Crawl-delay: 1
+Crawl-delay: 2
 Disallow: /api/
 Disallow: /applications/
 Disallow: /complete/