Why Page-Type Matters More Than Domain-Level Crawl Counts
Most crawl analytics dashboards report aggregate numbers: total bot requests per day, top crawlers, top countries. Useful, but blunt. The more interesting question for a content site is which kinds of pages get fetched the most. That answer reveals what AI crawlers think is worth re-checking, which in turn reveals what AI products are actually serving from those pages.
This page reports the answer for presenc.ai during April 2026, drawn from the same Cloudflare Worker plus D1 logging stack used for the rest of our crawl analytics research.
Fetch Frequency Ranking by Page Category
The table groups all logged AI crawler requests by URL pattern and reports the median number of fetches per unique URL during the month, broken down by major bot. Higher numbers mean more re-checking by that crawler.
| Page type | GPTBot | OAI-SearchBot | PerplexityBot | ClaudeBot | Google-Extended |
|---|---|---|---|---|---|
| Top-level hubs (/, /research, /compare) | Highest | Highest | Highest | High | High |
| Recent blog posts (last 30 days) | Very high | Very high | High | High | Moderate |
| /research/* pages | High | High | High | Moderate | Moderate |
| /compare/* | Moderate | High | High | Moderate | Moderate |
| /alternatives/* | Moderate | High | Highest | Moderate | Moderate |
| /guides/* | Moderate | Moderate | Moderate | Moderate | Low |
| /glossary/* | Low | Moderate | Low | Low | Low |
| Long-tail pSEO (/ai-platforms/{model}/{vertical}) | Low | Low | Very low | Very low | Very low |
| Static legal pages (/terms, /privacy) | Very low | Very low | Very low | Very low | Very low |
What the Pattern Tells Us
The strongest signal is that hubs and recent posts dominate AI crawler attention regardless of which bot is doing the fetching. This is consistent with how the underlying AI products work. ChatGPT search, Perplexity, and Claude with browsing all need fresh, well-indexed material to cite, and the cheapest way to find it is to recheck the same hub pages and recent posts repeatedly. Long-tail pSEO pages get a fraction of the attention even when they exist in the sitemap, which is exactly the "Discovered, currently not indexed" problem visible in Google Search Console for new pSEO clusters.
The second observation is that PerplexityBot allocates a disproportionate share of its budget to /compare and /alternatives URLs. This is what you would expect from a search engine that gets used heavily for tool-shopping queries. ClaudeBot, by contrast, shows the most bursty pattern, with re-fetches concentrated in 1 to 2 day windows that look like training or evaluation runs rather than continuous indexing.
The third observation is that OAI-SearchBot looks more like a search engine crawler than a training crawler. Its fetch frequency on hubs and recent posts approaches GPTBot levels, but its long-tail coverage is much shallower. That is consistent with the public statements that OpenAI uses OAI-SearchBot to feed its search index specifically.
Implications for Content Strategy
If you publish a lot of pSEO long-tail and want it picked up by AI products, hub-and-spoke linking matters enormously. The top-level hubs are crawled often. The spokes are not. Every spoke that does not get a strong link from an actively-crawled hub is invisible to the AI crawler economy. This is the same principle as traditional SEO internal linking, but the cost of getting it wrong is higher because the long tail of pSEO is precisely what most marketing teams are now investing in.
For brands that want to maximise live citation in browse-enabled AI assistants, the practical implication is to keep a small number of canonical, frequently-updated hubs on your most important topics, and rely on those hubs to feed crawler attention to recent and historical content alike. Update timestamps on hubs matter because AI crawlers track lastmod values to decide which URLs to recheck.
Methodology
Data drawn from the same Cloudflare Worker plus D1 logging stack described in our companion research on AI bots observed on presenc.ai. URL grouping is based on path-prefix matching against the canonical path patterns used in the pSEO registry. Median fetch counts are reported per category to dampen the influence of any single highly-fetched outlier URL. Period: 1 to 30 April 2026.
How Presenc AI Helps
Most teams do not have visibility into which of their pages get rechecked by AI crawlers and which do not. The deployment pattern shown here is a feature available to Presenc AI customers: a Cloudflare Worker that logs every request, classified by AI bot, with category-level fetch frequency reporting that makes the hub-and-spoke gap obvious. Combined with our standard AI visibility scoring, it shows both what AI is saying about you and where the crawl-budget gaps that limit that conversation actually are.