SecondDoorScanner

How it identifies itself

SecondDoorScanner/1.0 (+https://www.seconddoor.io/bot)

The browser pass announces itself as ordinary desktop Chrome instead. That is deliberate: it measures what a visitor sees, and sites that serve scanners a broken page would otherwise be scored on damage we caused rather than on their real front door. It obeys the same robots.txt rules as the address above, because the identity on the wire is a measuring instrument and not a claim about who we are.

One check asks whether your edge refuses the AI crawlers your robots.txt welcomes. It asks under our own name, carrying theirs:

SecondDoorScanner/1.0 (crawler-parity-probe; carrying the name GPTBot; +https://www.seconddoor.io/bot)

We do not send another company’s crawler identity. Earlier versions of this scanner sent the real GPTBot, ClaudeBot and PerplexityBot strings, and that was wrong: it borrows a name to obtain access we would not otherwise get. Carrying the name inside our own still trips any rule matching on it, which is the common case, and it means a refusal we report is a refusal we actually met. It also makes a pass weaker evidence than a block, so the report says so rather than claiming those crawlers are admitted.

What it fetches

• /robots.txt, /llms.txt, /sitemap.xml
• /.well-known/security.txt, mcp.json and ai-plugin.json
• Your homepage, once, for structured data and links
• Your pricing page, once as raw HTML and once rendered in a browser
• Up to two developer documentation pages linked from your homepage
• Your signup page, reached by following the primary call to action

A scan is roughly twenty requests. It never crawls beyond those pages, and it never submits a form, types into a field, or creates an account. Non-GET navigations are intercepted before they reach your server.

How it paces itself

• At least 300 milliseconds between requests to the same host.
• An eight second timeout per request.
• At most 400 kilobytes read from any single response.
• At most four redirects followed, each re-checked before it is opened.

What it obeys

• robots.txt is read for every host the scan touches, before anything else is requested from it, and resolved against the SecondDoorScanner group by longest match. A rule naming us beats a rule for everyone.
• Crawl-delay is honoured, up to two seconds. Above that a scan would run out of time and publish half a report, so it is refused instead.
• A path you disallow is recorded as declined, never as missing. A page we chose not to ask for is not a page you failed to publish, and no score moves because of it.
• Disallowing the whole site stops the scan before the homepage is read, and no report is written.
• Pages load their own stylesheets, scripts and images so the browser pass can render them. robots.txt governs which documents a crawler opens, and refusing a page its own assets would mean scoring you on damage we caused.
• If robots.txt cannot be read at all, because it times out or returns a challenge page, we retry once and then proceed. That is a deliberate departure from a strict reading of the standard: a wall you may not know is there should not quietly delete your site from the index.

How to block it

User-agent: SecondDoorScanner
Disallow: /

That rule stops the scanner. A scan of your site will end without reading the homepage and without writing a report, and the person who asked for it is told why.

Returning 403 to the user agent works too, though it is the blunter instrument: a scan then reports that automated access was refused, recorded as exactly that and never as the absence of whatever it was looking for.

If something about the scanner looks wrong from your side, write to support@seconddoor.io.