SecondDoor

Should you block AI agents? A straight answer

Blocking automation stops scrapers and card testers. It also stops the buyers arriving from ChatGPT and Claude. How to keep one without losing the other.

What blocking actually costs you now

Assistant shopping and assistant signup run inside a browser. When someone asks ChatGPT to find them a jacket, or asks Claude to sign them up for a trial, the thing that arrives at your site is a browser being driven by software. If your edge serves that an error page, the sale does not move elsewhere in your funnel. It moves to a competitor.

This is no longer a rounding error. Cloudflare measured bots at 57.4% of traffic to HTML content by June 2026, and Adobe recorded a 4,700% year on year rise in generative AI traffic to US retail sites. An independent crawl of 1,500 sites in January 2026 found roughly 30% blocking AI bots, and noted that most of those blocks looked accidental: inherited from a template, or a rule written years ago for a different problem.

The case for blocking is real

Scrapers lift your catalog and undercut you. Card testers hammer your checkout. Aggressive crawlers cost you bandwidth and skew your analytics. None of that goes away because agents got interesting, and anyone telling you to open everything up is not the one carrying your fraud losses.

So the question is not whether to defend yourself. It is whether your defence can tell the difference between a scraper and a customer's assistant.

Allow the named agents, keep the wall for everything else

The agents that carry real buyers announce themselves. ChatGPT-User, Claude-User, PerplexityBot and Google-Extended are documented, and they publish address ranges you can verify. Naming them in robots.txt and admitting them at the edge is a narrow, reversible change: everything unnamed still meets the defences you already have.

Verify rather than trust. A user agent string is a claim, not proof, and impersonating one is trivial. Match on published address ranges, or on the signed requests that the major agents increasingly send, and your allow-list admits the real ones while an impersonator gets the same challenge as before.

Then watch what happens. Segment the allowed agents in your logs and look at what they do. If a named agent behaves badly you can revoke it in one line, which is a much better position than never having found out what it would have bought.

Questions

Does allowing these agents mean my content gets used for training?
Not necessarily the same thing. Retrieval agents fetching a page to answer someone's question are distinct from training crawlers, and they are usually separate user agents you can allow or refuse independently. Decide the two separately rather than with one rule.
Will a Cloudflare or WAF challenge stop these agents?
Usually yes, and silently. A JavaScript challenge or CAPTCHA is exactly the wall an assistant cannot pass. If you run a challenge, add an exception for the verified agents rather than assuming they will get through.
How do I know if I am blocking them today?
Fetch your own product or signup page with a plain automated browser and see what comes back. If you get an error page, a challenge, or a near-empty shell, that is what an assistant sees. A SecondDoor scan reports this directly as browser access.
How machine discoverability is scoredCheck your own site
Should you block AI agents? A straight answer, SecondDoor