Разное → Блокировка кравлера ClaudeBot на уровне Apache

Пару дней назад появилась аномальная нагрузка на сервер. Анализ логов показал, что сайт сканирует некий ClaudeBot, причём сканирует довольно варварски - игнорирует robots.txt и шлёт десятки запросов в секунды с разных айпишников:

18.191.62.68 - - [23/Nov/2024:21:26:54 +0300] "GET /taxonomy/term/8%2C7%2C59%2C113%2C164 HTTP/1.0" 403 1516 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)"
3.12.154.133 - - [23/Nov/2024:21:26:54 +0300] "GET /taxonomy/term/35%2C57%2C147%2C59%2C7 HTTP/1.0" 403 1516 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)"
3.147.73.85 - - [23/Nov/2024:21:26:55 +0300] "GET /taxonomy/term/1%2C49%2C13%2C57%2C59 HTTP/1.0" 403 1516 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)"
13.59.205.182 - - [23/Nov/2024:21:26:55 +0300] "GET /taxonomy/term/1%2C207%2C7%2C29%2C45 HTTP/1.0" 403 1516 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)"
18.118.119.77 - - [23/Nov/2024:21:26:56 +0300] "GET /taxonomy/term/8%2C7%2C198%2C170%2C36 HTTP/1.0" 403 1516 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)"
18.191.237.228 - - [23/Nov/2024:21:26:56 +0300] "GET /taxonomy/term/1%2C130%2C92%2C118%2C91 HTTP/1.0" 403 1516 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)"
3.146.152.119 - - [23/Nov/2024:21:26:56 +0300] "GET /taxonomy/term/8%2C92%2C130%2C91%2C68 HTTP/1.0" 403 1516 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)"
18.218.108.24 - - [23/Nov/2024:21:26:57 +0300] "GET /taxonomy/term/1%2C207%2C57%2C7%2C33 HTTP/1.0" 403 1516 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)"
3.145.112.23 - - [23/Nov/2024:21:26:57 +0300] "GET /taxonomy/term/1%2C57%2C56%2C55%2C33 HTTP/1.0" 403 1516 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)"
18.219.207.115 - - [23/Nov/2024:21:26:57 +0300] "GET /taxonomy/term/8%2C12%2C119%2C7%2C126 HTTP/1.0" 403 1516 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)"
3.128.171.192 - - [23/Nov/2024:21:26:57 +0300] "GET /taxonomy/term/82%2C84%2C178%2C13%2C7 HTTP/1.0" 403 1516 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)"

Интернет говорит, что это бот компании Anthropic, который собирает данные для моделей искусственного интеллекта.

Баним с помощью .htaccess:

RewriteCond %{HTTP_USER_AGENT} ClaudeBot [NC]
RewriteRule .* - [F,L]