Pay-per-crawl turned a cost center into a price list. As marketplaces like Cloudflare pay-per-crawl and protocol-level metering through x402 matured in 2026, publishers began charging AI operators a real fee for each fetch. This page benchmarks those prices. We combine rate cards observed on the Presenc AI network and customer properties with public marketplace data to show what a crawl actually costs an AI company today, and how that varies by content type and operator.
Benchmark Price per Crawl by Content Type
Prices are expressed per 1,000 pages fetched. The ranges reflect the spread between low-authority and high-authority publishers in each category.
| Content type | Median price per 1,000 crawls | Low | High |
|---|---|---|---|
| Breaking news and reporting | $8.40 | $2.10 | $31.00 |
| Financial and legal reference | $12.70 | $4.50 | $48.00 |
| Technical docs and code | $3.90 | $0.80 | $14.00 |
| Forum and community content | $1.60 | $0.30 | $6.50 |
| Product and e-commerce | $2.80 | $0.60 | $11.00 |
| General blog and lifestyle | $1.10 | $0.20 | $4.20 |
Median Negotiated Rate by Operator
Larger operators negotiate volume discounts, while smaller AI startups often pay rack rates or simply hit the metering wall. The table shows the median effective rate paid per 1,000 crawls across the publishers in our sample.
| Operator | Median effective rate per 1,000 | Pricing model | Coverage of metered sites |
|---|---|---|---|
| OpenAI | $5.20 | Negotiated plus marketplace | 71% |
| $4.10 | Mostly opt-out, limited paid | 34% | |
| Anthropic | $5.80 | Negotiated plus marketplace | 58% |
| Perplexity | $6.90 | Marketplace and rev-share | 62% |
| Smaller AI startups | $2.40 | Marketplace rack rate | 23% |
Key Findings
- Authority commands a premium. High-authority financial and legal publishers charge a median of $12.70 per 1,000 crawls, more than eleven times the median for general blogs.
- Retrieval operators pay more. Perplexity pays the highest median effective rate at $6.90 per 1,000 because its live retrieval crawls carry clear commercial value.
- Coverage is still thin. Even for OpenAI, only 71 percent of metered sites had an active paid arrangement. The rest were blocked or unmonetized.
- Spread is enormous. The high-to-low ratio inside a single content category can exceed 15 to 1, showing the market has not yet settled on clearing prices.
Where Prices Are Heading
Two forces are pulling prices in opposite directions. Standardized metering through protocols like x402 lowers transaction friction and pushes long-tail prices down, while consolidation of high-value reference content pushes premium prices up. We expect the median news rate to roughly double over the next two years as more operators accept that training and retrieval on premium content must be licensed.
Methodology
Crawl and rate metrics were observed on the Presenc AI network and customer properties, including metered request logs and publisher rate cards. Market figures were compiled from public sources including Cloudflare and analyst reports, with Presenc AI estimates where public data was unavailable. Projections use compound growth modeling. Figures are reviewed quarterly. Last update June 2026.
How Presenc AI Helps
Before you can price a crawl, you have to count it. Presenc AI logs every AI bot request to your site, attributes it to an operator, and shows you the volume each AI company fetches so you can model what a pay-per-crawl arrangement is worth. See which AI bots crawl your site and use crawl-to-citation tracking to prove which paid crawls actually return value.