How should I use robots.txt to control AI crawler access?
Broadcastwell measures and fixes whether AI search engines name B2B software companies.
Run the free checkBroadcastwell's robots.txt, served on 2 November 2026, has separate Googlebot and Google-Extended groups, each with Allow: /. Choose the group for the crawler and use you intend to control, then set the relevant path rule. Our file records a declared access policy; it does not establish a visit, crawler compliance, training use or an answer citation. The provider's documentation is needed to identify each crawler's purpose.
Observed answers on stated dates, not a ranking.
What we measured
Measured. The Agent Gap task method recorded three public website tasks across 70 software vendors. It recorded 206 correct results, three incorrect and one unscored across 210 initial sessions. Thirty repeat sessions were separate. The study did not compare crawler policies, authenticate bot identities or test whether a rules change caused an answer to cite a page.
In the 6 October 2026 Agent Gap collection, all 210 initial tasks found a target page and 163 reached the defined endpoint.
Reported. Google's AI feature guidance, read 28 October 2026, identifies Googlebot rules as the crawl control for Search, including its AI features. Google-Extended governs different uses, including training and grounding in other Google products. Google's robots.txt introduction, read the same date, explains that crawler instructions depend on crawler cooperation and do not secure private content.
A separate description-task count
The Agent Gap protocol records reached endpoints in 67 of 70 initial Describe tasks on 6 October 2026. Those tasks concerned vendors' public websites. They did not identify the requests as OAI-SearchBot, GPTBot or an Anthropic robot, or test the effect of a robots rule. This description-task cut is part of the broader task collection above; it must not be added to that collection's endpoint total. A reached endpoint is not a verified crawler visit.
Provider purposes and the host being controlled
Reported. OpenAI's bot documentation, read on 29 October 2026, separates OAI-SearchBot for search from GPTBot for training. It describes ChatGPT-User as acting on user requests, separately from automatic search crawling. Anthropic's crawler article, also read on 29 October 2026, distinguishes ClaudeBot, Claude-SearchBot and Claude-User and gives their purposes and robots.txt instructions.
Anthropic distinguishes Claude-User's user-directed retrieval from Claude-SearchBot's search purpose. Its guide places robots.txt controls on the site or subdomain concerned. A documentation subdomain therefore needs its own addressed rule record, rather than an assumption that the homepage's file controls that separate host.
OpenAI's search help article, read on 29 October 2026, also asks publishers to allow traffic from its published searchbot IP (Internet Protocol) addresses at the host. A robots permission and a host restriction are separate checks; changing one does not verify the other. The bot's documented purpose, the declared policy and any actual observed visit remain distinct evidence.
Reasoned. If the intended policy restricts training while permitting search, inspect the separate named tokens. For an agency managing a documentation subdomain, record the intended purpose, exact bot and inspected address. A later answer that omits a company would need its own answer record; it would not identify which robot requested the page.
What this means for you
Reasoned. First write the access decision in plain language. You may want a public source available to search while choosing a different policy for another use. An undifferentiated instruction to block AI cannot express those separate decisions reliably.
Identify the provider's documented crawler token and the exact paths affected. Review the existing rules before adding a new group; a broad rule may cover more public content than the page you had in mind. Test representative allowed and disallowed URLs against the intended rule interpretation, then inspect the actual served robots.txt file.
Keep hosting and access controls in the same review. A page permitted by robots.txt can still be unavailable because of its server or delivery settings. A disallow instruction is also different from a request to remove a URL from search results. Use the provider's documented indexing or snippet controls for that separate goal.
Preserve a dated before-and-after rule record and the reason for the change. Verification should show which published rule applies to which URL. It cannot certify that every crawler obeyed it, that an assistant completed a task or that a future answer will include your company.
See the Manual’s citation chapter.
- For a site owner: separate search access from other provider uses.
- For a developer: test the named token and affected paths before changing the served file.
- For an agency: report policy delivery separately from observed requests or answers.
Also asked as
How do robots.txt rules affect AI crawler access?
What do AI crawlers do, and how can site owners control access?
Related
- Agent Gap's task boundaries
- Broader bot-access guide
- Google's crawl controls
- Special-file requirement question
- The Manual on exact events and measurement
- More fixes answers
Next step
Free 10-question check (one engine). No email is needed; runs are limited per network and per day, and by the site's daily allowance.
Our answer-study method, v1.1.
Answer Repair: $490.
Prepared by the Broadcastwell team.
Page owner: Rohan Iyer, AI Engineer. Corrections: hello@broadcastwell.com
Related evidence
- Does allowing GPTBot make a site appear in ChatGPT search?
- All fourteen category records
- Absence Index method and data definitions
Published . Last updated .