Add GPTBot to robots.txt

gptbot is very aggressive, let's disallow them along with the other very aggressive bots.
PTFS-Europe · Apr 24, 2024 · 4bc82a1 · 4bc82a1
1 parent 70ead5b
commit 4bc82a1
Showing 1 changed file with 3 additions and 0 deletions.
diff --git a/code/web/robots.txt b/code/web/robots.txt
@@ -80,6 +80,9 @@ Disallow: /
 User-Agent: MJ12bot
 Disallow: /
 
+User-Agent: GPTBot
+Disallow: /
+
 User-agent: 360Spider
 Disallow: /