What Is Claude-SearchBot?
Claude-SearchBot is the Anthropic crawler that indexes web content for Claude's search feature. Anthropic says it "navigates the web to improve search result quality for users" and "analyzes online content specifically to enhance the relevance and accuracy of search responses." It is one of three Anthropic agents, and it is the one that decides whether your pages can be found when Claude searches the web.
Anthropic's Three Agents
Anthropic clarified the roles of its crawlers in a documentation update dated February 20, 2026.
| Agent | Job | What blocking it does |
|---|---|---|
| ClaudeBot | Collects web content that could contribute to model training | Signals that the site's future content should be excluded from training datasets |
| Claude-SearchBot | Indexes content to improve Claude's search results | May reduce the site's visibility and accuracy in Claude's search results |
| Claude-User | Fetches a page when a Claude user asks a question that needs it | Stops Claude retrieving the page for that user, which may reduce visibility in user-directed search |
All three honour robots.txt, including the non-standard Crawl-delay extension. Each needs its own rule: blocking ClaudeBot does not block Claude-SearchBot or Claude-User. Anthropic also says it will not try to bypass CAPTCHAs, and that blocking by IP address may not work reliably as an opt-out.
robots.txt Examples
To opt out of training while staying visible in Claude's search and answers:
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /Why It Matters for AI Visibility
A ClaudeBot rule does not cover everything from Anthropic. A site that blocks all three is invisible to Claude's web search, so Claude answers questions about that brand from training data and third-party pages. A site that blocks only ClaudeBot keeps its content out of training and still gets cited. Claude's referral traffic grew sharply in 2026, which raises the cost of an accidental block: see Claude referral traffic growth and Claude citation patterns.
Commonly Confused With
Claude-SearchBot is the search-indexing counterpart to OAI-SearchBot, and Claude-User is the counterpart to ChatGPT-User. ClaudeBot is the training crawler, like GPTBot.