Should I block GPTBot and ClaudeBot?

Block them if you like: they gather training data, and saying no costs you little. What costs you is blocking the other bots from the same companies, the ones that decide whether you turn up in their answers. Most robots.txt files that block AI get this the wrong way round.

Three kinds of bot

Each of the big AI companies now runs more than one crawler, and they do different jobs:

• Training crawlers collect pages that may be used to train future models. Blocking one keeps your future pages out of that training. It doesn't remove anything already collected.
• Search crawlers build the index an assistant searches when it answers a question. Block one and that assistant can't cite you.
• User fetchers load a page because someone asked the assistant to read it, there and then. They aren't crawling, and OpenAI says robots.txt may not apply to them.

As the companies document them today:

* Not a crawler: a name you put in robots.txt to say how content the ordinary crawler already fetched may be used. Google says blocking Google-Extended doesn't affect Google Search or rankings. Common Crawl publishes a free archive of the web that many AI training sets are built from, so CCBot belongs with the training crawlers.

A robots.txt that does both

If you'd rather not be training data but do want to be found, block the training names and leave the rest alone:

# Stay out of AI training
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: CCBot
Disallow: /

# Everyone else, AI search included
User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

A crawler obeys the most specific group that names it, so OAI-SearchBot and Claude-SearchBot fall through to * and get in. Keep any Disallow lines you already have for admin or private paths in the * group.

Things robots.txt can't do

• It's a request. The companies above say they honour it. Not every crawler does, and a scraper that wants your pages won't ask.
• It isn't retroactive. Blocking today doesn't pull your pages out of a model trained last year.
• It isn't the only switch. Hosts and CDNs have their own bot blocking. Cloudflare can block AI crawlers before they ever read your robots.txt, so check your host's settings as well as the file.
• It doesn't hide anything. A Disallow line is public, and it lists the paths you didn't want looked at. Private pages need a login, not a robots.txt entry.

So, should you?

For most small products, the question that matters is the search one, and the answer is to let those bots in: being quotable is how a new app gets mentioned at all. Whether to block training is a matter of taste, not of visibility. If you sell writing, art or anything else that is the content itself, blocking training crawlers is a reasonable line to draw. If you sell software, being in the training data does little for you either way.

The visibility check reports which AI crawlers your robots.txt blocks and doesn't score it, because either choice is fair. It does flag a block on the search engines themselves, which is almost always a mistake.