What this is
AI crawler volume is the most under-discussed macro shift in the open web. Four years ago AI bots were a rounding error; today they account for a fifth of all bot traffic and are on a trajectory that crosses human traffic before the end of 2027. This page is a 2026-05-15 snapshot of how we got here, year by year.
AI Bot Share of Total Crawler Traffic
| Date | AI bot share of crawler traffic | AI bot share of all bot traffic | Dominant bot |
|---|---|---|---|
| 2022 | <1% | ~2% | CCBot |
| Jan 2023 | ~1.2% | ~4% | CCBot |
| Aug 2023 | ~1.8% | ~6% | GPTBot (launched Aug 2023) |
| May 2024 | ~2.7% | ~10% | Bytespider (42% AI share) |
| May 2025 | ~4.1% | ~16% | GPTBot (30% AI share) |
| Jul 2025 | ~4.5% | ~18% | GPTBot (11.7%) |
| Q1 2026 | 5.06% | 22% | Meta-ExternalAgent (16.7%) |
Year-over-Year Bot Growth (May 2024 to May 2025)
| Bot | YoY change | Notes |
|---|---|---|
| GPTBot | +305% | Biggest absolute mover |
| Googlebot | +96% | Reflects AI Overviews |
| ClaudeBot | +64% | Steady ramp |
| Meta-ExternalAgent | +~700% | From near-zero baseline |
| Bytespider | -83% | Geopolitical blocking |
| Applebot | +124% in a single month | Apple Intelligence ramp |
Q1 2026 Bot Market Share (of AI bot traffic only)
| Rank | Bot | Share of AI bot traffic |
|---|---|---|
| 1 | Googlebot (AI portion via Google-Extended) | ~22% |
| 2 | Meta-ExternalAgent | 16.7% |
| 3 | GPTBot | 12.0% |
| 4 | ClaudeBot | 11.7% |
| 5 | Bytespider | 7.1% |
| 6 | Applebot | 5.8% |
| 7 | PerplexityBot | 4.2% |
| 8 | OAI-SearchBot | 3.4% |
Six Things the Data Tells You
- The crossover is coming, not happening yet. Bots are 31.2% of all HTTP requests in early 2026; the trajectory crosses human traffic before end of 2027.
- GPTBot was the singular driver of 2024-2025 growth. +305% YoY is unprecedented for an established crawler category.
- Meta-ExternalAgent went from nothing to second place. Llama 4 training pulled an enormous amount of fresh data in Q4 2025 and Q1 2026.
- Bytespider collapsed. Geopolitical blocking dropped its share from 42% in May 2024 to 7% by Q1 2026.
- Applebot is the new mover. +124% in a single month coincided with the Apple Intelligence rollout to all EU markets.
- Dedicated training crawlers now generate ~50% of AI bot traffic. The other half is answer-engine and agent crawling, which is the part that drives visible citations.
What This Means for AI Visibility
The AI-bot category is no longer monolithic. Training crawlers (GPTBot, ClaudeBot when configured for training, CCBot, Meta-ExternalAgent) and answer-engine crawlers (OAI-SearchBot, PerplexityBot, Google-Extended for AI Overviews) are diverging in volume, intent, and citation-yield. Site owners who block both indiscriminately are also blocking measurable visible traffic, while site owners who allow both indiscriminately are leaking training data with no return. The right 2026 policy is granular: allow what cites, block what only trains.
Methodology
Volume figures are taken from the Cloudflare Crawl-to-Click report, the Cloudflare Radar 2025 Year in Review, the Cloudflare deeper look at AI crawler traffic by purpose, and WebSearch API's March 2026 monthly report. Q1 2026 market share is cross-checked against Technology Checker's billions-of-requests analysis.
How Presenc AI Helps
Presenc AI tracks AI bot volume on your specific domain against industry and Cloudflare-aggregate baselines, so you can see whether GPTBot, Meta-ExternalAgent, or Applebot is over- or under-crawling you relative to your size and content depth. The dashboard separates training-only bots from answer-engine bots, which is the only useful axis for an allow/disallow decision in 2026.