AI Crawler User Agents: Complete 2026 Reference

Why AI Crawler User Agents Matter

AI crawlers are the automated agents that large language model operators send across the web to collect training data, retrieve content for real-time answers, and index pages for AI-powered search. Unlike traditional search engine bots that have been operating for decades, AI crawlers are a relatively new category — and they are multiplying fast. Understanding exactly which bots are visiting your site, what they want, and whether they respect your access rules is now a core requirement for any brand managing its AI visibility.

Every HTTP request from a crawler includes a user agent string — a text identifier that tells your server who is making the request. By recognizing AI crawler user agents, you can monitor crawl activity, enforce access policies via robots.txt, analyze which AI platforms are indexing your content, and make strategic decisions about which crawlers to allow or block. This reference provides the complete list of known AI crawler user agents as of early 2026, along with practical guidance on detection, configuration, and monitoring.

Complete AI Crawler User Agent Reference Table

The following table lists all major AI crawler user agents currently active on the web. Each entry includes the bot name, the exact user agent string to match, the operating company, the crawler's purpose, whether it honors robots.txt directives, and when it was first widely observed.

This table is maintained by the Presenc AI research team and updated as new AI crawlers emerge. The AI crawler ecosystem is evolving rapidly — new bots appear frequently, and existing bots update their user agent strings and behaviors. Always verify against the latest documentation from each operator.

How to Identify AI Crawlers in Server Logs

Your web server logs record the user agent string for every request. To identify AI crawler activity, you can use regex patterns to match known AI bot signatures. Here are practical patterns for the most common log formats:

Combined regex pattern for all major AI crawlers:

GPTBot|ChatGPT-User|OAI-SearchBot|ClaudeBot|anthropic-ai|PerplexityBot|Google-Extended|GoogleOther|Bytespider|CCBot|Amazonbot|FacebookBot|AppleBot-Extended|cohere-ai|meta-externalagent

Example: grep command to extract AI crawler requests from an Apache/Nginx access log:

grep -E "GPTBot|ChatGPT-User|OAI-SearchBot|ClaudeBot|anthropic-ai|PerplexityBot|Google-Extended|GoogleOther|Bytespider|CCBot|Amazonbot|FacebookBot|AppleBot-Extended|cohere-ai|meta-externalagent" /var/log/nginx/access.log

Example: AWK command to count requests per AI crawler:

awk -F'"' '/GPTBot|ChatGPT-User|OAI-SearchBot|ClaudeBot|PerplexityBot|Google-Extended|Bytespider|CCBot|Amazonbot|FacebookBot|cohere-ai|meta-externalagent/ {for(i=1;i<=NF;i++) if($i ~ /GPTBot|ChatGPT-User|OAI-SearchBot|ClaudeBot|PerplexityBot/) print $i}' access.log | sort | uniq -c | sort -rn

For more sophisticated analysis, consider piping log data into a structured analytics tool or using Presenc AI's built-in crawler monitoring, which automatically classifies and tracks AI bot activity on your site.

Robots.txt Configuration for AI Crawlers

The robots.txt file remains the primary mechanism for controlling AI crawler access. Each AI crawler uses a specific token that you can target with Allow or Disallow directives. Here are common configuration patterns:

Block all AI training crawlers but allow retrieval bots:

# Block training crawlers
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

# Allow retrieval/search bots
User-agent: ChatGPT-User
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

Allow all AI crawlers (maximize AI visibility):

User-agent: GPTBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

Selective access — allow crawling of public content only:

User-agent: GPTBot
Allow: /blog/
Allow: /docs/
Allow: /resources/
Disallow: /

User-agent: ClaudeBot
Allow: /blog/
Allow: /docs/
Allow: /resources/
Disallow: /

Important: robots.txt is a voluntary standard. While all major AI operators listed above respect robots.txt directives (with the exception of partial compliance from Bytespider), enforcement ultimately depends on the crawler operator's compliance. There is no technical mechanism to prevent a non-compliant crawler from accessing your content.

Training Crawlers vs. Retrieval Crawlers

Understanding the distinction between training crawlers and retrieval crawlers is essential for making informed access decisions:

Training crawlers (e.g., GPTBot, ClaudeBot, Google-Extended, CCBot) collect web content that is later used to train or fine-tune AI models. Once your content is ingested into training data, it becomes part of the model's learned knowledge. Training crawls are typically large-scale, infrequent (relative to retrieval), and the data is used long after collection. Blocking a training crawler means future model versions will not learn from your content, which can reduce your brand's knowledge presence in AI systems.

Retrieval crawlers (e.g., ChatGPT-User, OAI-SearchBot, PerplexityBot) fetch content in real time to augment AI responses via retrieval-augmented generation (RAG). When a user asks a question, the AI system retrieves relevant web pages and uses them to generate a cited, up-to-date answer. Blocking a retrieval crawler means your content will not appear in real-time AI search results on that platform, directly impacting your visibility for live queries.

Some crawlers serve dual purposes, and operators may introduce new user agents over time. The strategic decision of which crawlers to allow depends on your brand's AI visibility goals. Brands focused on maximizing AI presence typically allow both training and retrieval crawlers. Brands with data licensing concerns may block training crawlers while allowing retrieval bots to maintain search visibility.

Monitoring AI Crawler Activity

Passive log analysis tells you which AI crawlers have visited your site, but proactive monitoring is necessary to understand the full picture:

• Crawl frequency tracking: Monitor how often each AI crawler visits your site and which pages they request most frequently. Changes in crawl frequency can signal shifts in how AI platforms prioritize your content.
• Page coverage analysis: Identify which pages AI crawlers are accessing and which they are skipping. Gaps in crawl coverage may indicate technical issues (JavaScript rendering, slow load times) that prevent AI bots from accessing your content.
• Crawl budget impact: AI crawler traffic adds to your server's total crawl load. Monitor whether AI crawlers are consuming excessive resources or competing with traditional search engine crawlers for bandwidth.
• New bot detection: Set up alerts for user agent strings that match AI-like patterns but do not correspond to known bots. New AI crawlers appear regularly, and early detection helps you make access decisions before significant content has been scraped.
• Response code monitoring: Track whether AI crawlers are receiving 200 (success), 403 (forbidden), or 429 (rate limited) responses. High error rates may indicate misconfiguration or server-side issues affecting your AI visibility.

How Presenc AI Tracks AI Crawler Behavior

Presenc AI provides automated AI crawler monitoring as part of its visibility platform. Rather than manually parsing server logs, Presenc integrates with your site analytics and server data to identify all AI crawler activity, classify each bot by operator and purpose, track crawl trends over time, and alert you to new or unusual crawler behavior. Combined with Presenc's AI response monitoring — which tracks what AI platforms actually say about your brand — crawler monitoring completes the picture: you can see both which AI systems are accessing your content and how that content is being used in AI-generated responses. This end-to-end visibility enables data-driven decisions about crawler access policies and content optimization for AI platforms.