Research

AI Agents & Crawl Economics

Research on agentic browsing, AI shopping agents, crawler behaviour, and the economics of AI crawling. Agent decision patterns, x402, payment rails, and what crawlers actually fetch.

Agents now browse, evaluate, and transact on behalf of users, and crawlers now carry real cost. These reports cover how agents pick brands and what the crawl economy is doing to publishers.

How Fast AI Crawlers Index New Pages

First-party data on how quickly GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot discover and crawl new pages after deployment. Measured across 300 pSEO pages.

GPTBot Crawl Behavior in 2026

Deep dive into GPTBot crawl patterns: peak times, page size preferences, re-crawl rates, and content type prioritization based on first-party Cloudflare data.

AI Crawler Page Size Preferences

Analysis of how page size affects AI crawler behavior. First-party data on crawl frequency, re-crawl rates, and discovery speed across different page weights.

ChatGPT-User Bot: What It Means for Your Site

What ChatGPT-User requests mean, how to track them, and which pages get cited. Explains the three OpenAI bots and their differences.

AI Crawler User Agents: Complete 2026 Reference

Complete reference list of AI crawler user agent strings for GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, GoogleOther, and more. How to identify and manage AI crawlers.

AI Agents Are Making Crypto Decisions. Is Your Protocol Visible?

Research on AI agents executing autonomous on-chain transactions and how protocol AI visibility affects agent-mediated volume. Data on agent market share, routing decisions, and visibility factors.

How AI Agents Choose Brands: Decision Factors and Visibility Data

Research into how AI agents evaluate and recommend brands. Covers the decision factors, data sources, and optimization strategies for agentic brand visibility.

AI Crawler Block Rate by Industry 2026

Industry-level data on how often GPTBot, PerplexityBot, ClaudeBot, and Google-Extended are blocked via robots.txt. Benchmark your sector against cross-industry averages.

AI Bots Observed on Presenc AI in April 2026

A first-party log analysis of every AI crawler that hit presenc.ai in April 2026, captured by our Cloudflare Worker and D1 store. User agents, request volumes, fetch frequencies, and what changed since March.

AI Crawler Fetch Frequency by Page Type

How often AI crawlers re-fetch different page types on a single domain, measured from first-party Cloudflare Worker logs in April 2026. Hubs vs spokes, research vs guides, blog vs product, and what the variance tells us about crawl prioritisation.

AI Crawler Time-of-Day Patterns in 2026

When AI crawlers actually fetch your site, hour by hour. First-party data from a Cloudflare Worker logging every inbound request to presenc.ai, with implications for serving strategy, edge caching, and origin sizing.

Most-Fetched Pages by AI Bots in April 2026

The single URLs that AI crawlers re-fetched most often during April 2026 on presenc.ai, with attribution by bot. What it tells us about which content AI products actually use, and why ranking by fetch count is more useful than ranking by impressions.

OpenAI Operator Brand Visibility in 2026

How OpenAI Operator chooses which brands and which sites to interact with on behalf of users. Selection patterns, the role of structured product data, the interplay with ChatGPT search, and what brands need to do to be visible in agent-driven flows.

Agentic Commerce Market Readiness in 2026

How ready the commerce stack actually is for AI agents in 2026: payment authorization, identity, schema coverage, anti-bot infrastructure, and the gap between agent-side capability and merchant-side instrumentation. A practical readiness scorecard.

Cloudflare Pay-Per-Crawl: State of the Marketplace in 2026

Where Cloudflare Pay-Per-Crawl actually stands in April 2026: enrolled customers, daily HTTP 402 volumes, AI-side adoption, pricing distribution, and what the data tells us about the market.

AI Shopping Agents Brand Visibility 2026

How AI shopping agents (OpenAI Operator, Anthropic Computer Use, Claude Agent SDK, Gemini Deep Research) decide which brands to purchase from. Authorisation flows, agent-readable signals, and the brand-visibility patterns that determine inclusion in agent-driven commerce.

Agentic Browsing Brand Visibility 2026

Cross-agent analysis of agentic browsing brand visibility. How Operator, Computer Use, Gemini Deep Research, Grok 4 agentic, and Comet agentic mode evaluate brands during multi-step research and transaction tasks.

Agent Purchase Authorization Flows 2026

Synthesis of the agent-payment authorization frameworks, Visa TAP, Mastercard Agent Pay, AP2, x402, MPP, and how they shape brand visibility in agent-mediated commerce in 2026.

Browser AI Agents Brand Visibility 2026

How AI-native browsers (Perplexity Comet, ChatGPT Atlas, Dia, Arc) are reshaping brand visibility. Inline page summaries, address-bar AI search, and agentic-task brand inclusion patterns.

AI Agent Failure-Mode Statistics 2026

Why AI agent pilots stall in 2026: failure-mode decomposition (memory, tool error, hallucinated state, timeout), pilot-to-production conversion rates, and what separates successful deployments.

AI Agent ROI by Use Case 2026

Outcome-based ROI data for AI agent deployments in 2026 across sales SDR, customer support, internal IT, code generation, recruiting, and operations. Hours saved, revenue generated, deployment cost.

AI Agent Taxonomy 2026

A practical taxonomy of AI agents in 2026. Capability tiers (chatbot, copilot, tool-orchestrating, autonomous, multi-agent), with concrete examples per tier, decision criteria, and benchmark surrogates.

Workflows vs Agents: A Capability Matrix

Most products labelled "AI agents" in 2026 are actually workflows. A capability matrix to tell them apart: tool use, planning, memory, error recovery, autonomy. Honest classification of leading vendor products.

Multi-Agent Orchestration Frameworks 2026

Honest comparison of multi-agent orchestration frameworks in 2026: LangGraph, CrewAI, Microsoft AutoGen, OpenAI Swarm, Google ADK, Anthropic Skills compositions. Production-readiness, ergonomics, ecosystem.

Vertical AI Agent Market Map 2026

Landscape map of vertical AI agents in 2026 across sales, customer support, legal, finance, recruiting, healthcare, and operations. Notable vendors per vertical, deployment maturity, and where the market is consolidating.

Enterprise AI Agent Buying Criteria 2026

What enterprise procurement teams actually weigh when buying AI agents in 2026. Top criteria, weighted scores by company size, and how vendor positioning aligns (or fails to align) with buyer priorities.

MCP Server Ecosystem Statistics 2026

Adoption and ecosystem statistics for the Model Context Protocol (MCP) in 2026: server directory growth, top servers by usage, client adoption (Claude Desktop, Cursor, OpenAI), and the brand-visibility implications of an open AI tool standard.

AI Bot Share of Web Traffic April 2026

Cloudflare Radar data on AI bot traffic share in April 2026. Googlebot, Meta-ExternalAgent, GPTBot, ClaudeBot, Applebot ranked by request share, with training-crawler share now at 49.9 percent of all AI bot traffic.

OpenAI Operator Update Tracker, May 2026

Where OpenAI Operator went in 2026: shutdown August 31 2025, absorbed into ChatGPT Agent, and the current state of OpenAI computer-using agent capabilities including CUA model benchmarks and competitive landscape.

Zero-Click AI Search Agent Optimization, 2026

How brands optimize for AI agents that produce zero-click answers in 2026. Strategies for ChatGPT, Claude, Gemini, Perplexity, and AI agents that answer without sending users to your site.

A2A vs MCP, Agent Communication Standards in 2026

Side-by-side comparison of Google's Agent-to-Agent (A2A) protocol and Anthropic's Model Context Protocol (MCP) as the two foundational standards of the agentic AI stack in 2026. Adoption, governance, and architectural fit.

The Agentic Browser Wars 2026

Side-by-side comparison of the agentic browsers competing in 2026: ChatGPT Atlas, Perplexity Comet, The Browser Company Dia, Arc, and Brave Leo. Agent capabilities, market positioning, and the brand-visibility implications.

AI Agent Payment Standards Adoption, May 2026

Adoption tracker for AI agent payment standards in 2026: Coinbase x402, Google Agent Payments (AP2), Visa TAP, Microsoft MPP, and Agent Pay. Live integrations, transaction volumes, and the standards consolidation picture.

AI Agent Marketplaces Landscape, May 2026

How AI agent marketplaces are consolidating in 2026. Microsoft Marketplace, Salesforce Agentforce, Google Gemini Enterprise, ChatGPT Workspace Agents, Claude Skills directory, and the 1.2M+ open Agent Skills ecosystem.

Claude Skills and Subagents Ecosystem, May 2026

Anthropic Claude Skills (Dec 2025 launch) and subagent hierarchies in 2026. Open standard adoption, OpenAI Codex compatibility, 1.2M+ available skills, and the brand-visibility implications of skill-mediated tasks.

Marketing to AI Agents: The New Discipline, May 2026

How marketing to AI agents differs from marketing to humans in 2026. Agent-readable content, MCP servers as marketing channels, structured product feeds, and the surfaces where brands must show up to be considered by agent buyers.

Agent SEO: Optimizing Content for AI Agent Consumption

Concrete tactics for optimizing web content for AI agent extraction in 2026. JSON-LD schema, llms.txt, agent-readable changelogs, FAQPage structure, prose-first formatting, and code-fence patterns that maximise agent-tool-call success.

Agent-Readable Product Feeds, May 2026

How brands should structure product feeds for AI agent consumption in 2026. Google Merchant Center, Schema.org Product, ChatGPT Shopping feeds, Perplexity product cards, and emerging agent-feed standards compared.

Business-to-Agents (B2A) Commerce Models, May 2026

How businesses are adapting to AI agents as customers in 2026. B2A pricing models, agent-aware authentication, agent-tagged invoicing, B2A-specific terms of service, and the structural differences from B2B and B2C.

API Pricing for AI Agents vs Humans, May 2026

How major SaaS vendors are tiering API pricing for AI agent consumption in 2026. Stripe, Salesforce, Slack, HubSpot, Shopify, Microsoft, Atlassian, and the structural differences from human-developer API pricing.

Verified Agent Programs Across Platforms, May 2026

The emerging "blue checkmark for AI agents" landscape in 2026. Visa Trusted Agent, OpenAI Verified Agents, Anthropic Trusted Computer Use, Stripe verified payment agents, Salesforce Trusted Agents, and what verification gets you.

Agent-to-Agent Negotiation Patterns, May 2026

Catalog of how AI agents negotiate price, capability matching, and delegation in 2026. AP2 mandate patterns, x402 negotiation flows, A2A task envelopes, and the production patterns emerging from multi-agent deployments.

Agent Reputation and Identity Layer, May 2026

How AI agents prove identity and accumulate reputation in 2026 multi-agent systems. Verifiable credentials, on-chain agent IDs, AP2 mandate signing, AAIF agent registries, and the protocol-level identity/reputation infrastructure.

Cross-Agent Payment Flows, May 2026

How money moves when agent A hires agent B that uses agent C in 2026. Multi-hop settlement, escrow, dispute, and chargeback patterns built on AP2, x402, and Stripe agent-toolkit primitives.

AI Agent Infrastructure Startups Landscape, May 2026

The funded startups building horizontal infrastructure for AI agents in 2026: Temporal, Inngest, Braintrust, LangSmith, Helicone, Langfuse (acquired by ClickHouse), AgentOps, Restate. Funding rounds, valuations, and category positioning.

AI Agent Observability Startups, May 2026

Side-by-side comparison of AI agent observability platforms in 2026. LangSmith, Helicone, Braintrust, Langfuse (ClickHouse), AgentOps, Arize Phoenix, Galileo, Latitude. Feature matrix, pricing, and category positioning.

AI Agent Evaluation and RLHF Startups, May 2026

The funded startups building evaluation, RLHF, and data-labelling infrastructure for AI agents in 2026. Scale AI, Surge AI, Toloka, Argilla, Patronus AI, Confident AI, and the agent-eval supply chain.

AI Agent Identity and Auth Startups, May 2026

Startups building agent identity, authentication, and authorisation infrastructure in 2026. Auth0 agent extensions, Stytch, WorkOS, Stainless, Anrok, Curity, Plaid for AI agents, and the emerging agent-identity layer.

Agentic Commerce Startups, May 2026

Startups building agentic commerce infrastructure in 2026. x402-ecosystem companies, AP2 integrators, agent-checkout platforms, agent-side wallets, and the payment-routing layer for AI-mediated purchases.

State of Robots.txt for AI, May 2026

How websites are using robots.txt to block AI crawlers in 2026. 25 percent of top 1000 sites blocking GPTBot, ClaudeBot the fastest-growing block target, and the middle-path strategy that lets search bots through while blocking training crawlers.

AI Crawler Behavior on the Top 1,000 Sites 2026

How GPTBot, ClaudeBot, PerplexityBot, and CCBot crawl the top 1,000 websites in 2026. 25% block GPTBot, ClaudeBot crawls 20,583 pages per referral. Snapshot for 2026-05-15.

AI Crawler Volume Growth 2022-2026

AI bots grew from <1% of web traffic in 2022 to 22% of all bot traffic by 2026. GPTBot +305% YoY in 2025. Dedicated training crawlers now generate 49.9% of AI bot traffic.

AI Agent Engineer Career Guide 2026

Agentic AI engineers earn $185K-$320K base ($300K+ top); LangChain in 34.3% of postings; 60% of new enterprise projects include agents. Snapshot for 2026-05-15.

x402 Protocol Adoption Tracker 2026

x402 protocol 2026: 167M settled transactions but ~50% gamified, $28K daily real volume. x402 Foundation includes Cloudflare, Stripe, AWS, Google, Visa, Circle. Snapshot for 2026-05-15.

AI Coding Agent Market 2026

Cursor $2B ARR with 7M MAU, Cognition $25B valuation talks, Claude Code 18% workplace adoption, GitHub Copilot 4.7M paid users at 42% share. The 2026 AI coding agent market dissected.

Enterprise Agentic AI Adoption 2026

Enterprise agentic AI in 2026: 160k+ organisations deployed 400k+ Copilot Studio agents, Salesforce Agentforce 29k deals at $800M ARR, Microsoft 365 Agent 365 GA, plus the broader vendor landscape.

Google Search Agents and Brand Visibility (I/O 2026)

How Google Search Agents announced at Google I/O 2026 reshape brand discovery with always-on AI monitoring, recurring impressions, and synthesized topic updates.

Information Agents in Google Search (I/O 2026)

Google I/O 2026 introduced Information Agents, the first Search Agent type that monitors topics and pushes synthesized updates. What this means for brand inclusion and GEO.

Daily Brief: Gemini's Out-of-the-Box Agent (I/O 2026)

Daily Brief is Gemini's new out-of-the-box agent at Google I/O 2026 that synthesizes inbox, calendar, and tasks into a personalized daily digest with next-step suggestions.

Managed Agents in the Gemini API (I/O 2026)

Managed Agents provisions remote Linux environments for Gemini agent reasoning and tool execution. Analyze infrastructure and developer brand implications from Google I/O 2026.

WebMCP: A Proposed Web Standard for Browser Agents (I/O 2026)

WebMCP lets websites declare structured tools for browser agents, making sites agent-actionable. Explore web standard and brand visibility implications from Google I/O 2026.

AI Shopping Agent Conversion Statistics 2026

2026 conversion statistics for AI shopping agents vs human shoppers, by platform and category. Modeled from observed agent traffic and vendor benchmarks.

Agent Checkout Success Rate Benchmarks 2026

2026 benchmarks for AI agent checkout success rates and the top failure causes. Modeled from observed agent checkout attempts and vendor data.

Agentic Commerce GMV Forecast 2026-2030

Agentic commerce GMV forecast for 2026 to 2030, by region and category. Modeled from observed agent traffic and compound growth projections.

Which Retailers Are Agent-Payable in 2026?

2026 study of top retailers' readiness for AI agent checkout and payment rails. Modeled from observed agent sessions and public commerce signals.

Agent Cart Abandonment Statistics 2026

2026 statistics on AI agent cart abandonment, where agents drop off and why. Modeled from observed agent sessions and vendor benchmarks.

AI Agent Product Discovery Patterns 2026

How AI agents find and shortlist products in 2026, and which signals they weight. Modeled from observed agent sessions and public discovery data.

Agent-Payable Readiness by Industry 2026

2026 agent-payable readiness scores across industries, with gaps and leaders by sector. Modeled from observed agent sessions and public signals.

AI Crawler Market Share by Bot 2026

Which AI bots dominate crawl volume in 2026. GPTBot, ClaudeBot, PerplexityBot, Google-Extended and Bytespider share, observed on the Presenc AI network.

Pay-Per-Crawl Pricing Benchmarks 2026

What publishers charge AI crawlers per page in 2026. Pay-per-crawl pricing benchmarks across marketplaces, by content type and operator, with observed rates.

AI Bot Traffic Cost to Publishers 2026

What AI crawling actually costs publishers in 2026. Bandwidth and infrastructure cost of AI bots versus the referral value they return, modeled per million pageviews.

AI Crawl-to-Click Ratio by Platform 2026

How many pages AI platforms crawl per referral click they send back in 2026. Crawl-to-click ratios for OpenAI, Google, Anthropic and Perplexity, observed first-party.

AI Search Market Size Forecast 2026-2030

AI search market size and revenue forecast for 2026 to 2030. Multi-year projections by segment and region with modeled CAGR, from Presenc AI research.

GEO Market Size and Spend Forecast 2026-2030

Generative engine optimization market size and spend forecast for 2026 to 2030. Multi-year GEO tools and services spend by segment, with modeled CAGR.

AI Assistant Penetration by Country 2026

AI assistant adoption by country in 2026. Penetration rates across major markets, growth trends and platform mix, from Presenc AI research and public data.

Agentic Checkout Conversion Benchmarks

How often agent-initiated purchases actually complete in 2026. Completion rates by protocol maturity, where agent checkouts fail, how the funnel differs from human checkout, and what merchants can instrument.

Stablecoin Rails for Agent Payments

Why agent payments settled first on stablecoins rather than cards. Settlement properties that suit machine transactions, where the card rails are catching up, and the remaining gaps in the stablecoin path.

AI Agent Liability and Insurance

Who is responsible when an autonomous agent buys the wrong thing, leaks data, or causes loss. Liability allocation across principal, developer, and platform, emerging insurance products, and the contractual gaps.

Track Your AI Visibility

See how your brand appears across ChatGPT, Claude, Perplexity, and other AI platforms. Start monitoring today.