Robots.txt

1 post
Websites are Blocking the Wrong AI Scrapers (Because AI Companies Keep Making New Ones)
AI

Websites are Blocking the Wrong AI Scrapers (Because AI Companies Keep Making New Ones)

Hundreds of sites have put old Anthropic scrapers on their blocklist, while leaving a new one unblocked.