Research Overview
llms.txt is the emerging convention for declaring how AI systems should interact with your content, analogous to robots.txt for search engines but with richer directive semantics. Adoption is still early, but the shape of the early-adopter curve already reveals which industries are treating AI access as a strategic lever and which are not. This report presents the first cross-industry llms.txt adoption benchmarks, audited from a representative sample of domains per sector.
Adoption Metrics by Industry
For each of 15 industries, we measured four dimensions: overall adoption rate (percentage of audited domains serving a valid llms.txt), directive density (average number of Allow, Disallow, and metadata directives per file), Allow-to-Disallow ratio (a permissiveness signal), and adoption rate within the top 100 domains of that sector.
| Industry | Adoption % | Avg Directives per File | Allow:Disallow Ratio | Top-100 Adoption % |
|---|---|---|---|---|
| Technology / SaaS | 24 | 14 | 2.3:1 | 61 |
| Media / Publishing | 11 | 22 | 0.6:1 | 38 |
| Education / EdTech | 18 | 12 | 1.9:1 | 47 |
| Healthcare | 6 | 9 | 1.2:1 | 22 |
| Financial Services | 8 | 18 | 0.8:1 | 31 |
| E-Commerce / Retail | 9 | 11 | 1.5:1 | 34 |
| Cybersecurity | 21 | 16 | 1.7:1 | 52 |
| Legal | 5 | 15 | 0.7:1 | 18 |
| Real Estate | 4 | 8 | 1.8:1 | 15 |
| Travel / Hospitality | 7 | 10 | 1.6:1 | 24 |
| Automotive | 6 | 9 | 1.4:1 | 21 |
| Insurance | 4 | 13 | 0.9:1 | 14 |
| Food & Beverage | 5 | 8 | 1.7:1 | 17 |
| HR / Recruiting | 10 | 10 | 1.5:1 | 29 |
| Blockchain / Crypto | 27 | 13 | 2.4:1 | 64 |
Blockchain and technology lead adoption, reflecting technically aware teams who ship infrastructure conventions early. Media and legal publish llms.txt files more reluctantly but, when they do, use significantly more directives per file and skew toward Disallow. That pattern signals defensive intent rather than engagement: the file is being used as a gatekeeping layer, not an invitation.
Top-100 Adoption vs Long-Tail Adoption
Every industry shows a gap between overall adoption and top-100 adoption, but the size of the gap varies. In technology, top-100 adoption is 2.5x the overall rate, meaning the largest brands are moving first and the long tail is catching up. In real estate and insurance, the gap is nearly 4x, suggesting that the practice is concentrated among a small group of sophisticated brands while the typical domain has not yet engaged. Sectors with large top-100 gaps represent the clearest opportunity for mid-sized brands to establish a competitive advantage through early llms.txt adoption.
Allow-to-Disallow as a Posture Signal
The Allow-to-Disallow ratio is a useful posture signal. Ratios above 1:1 indicate a permissive posture that invites AI engagement with explicit consent. Ratios below 1:1 indicate a restrictive posture that uses llms.txt primarily to carve out exceptions to a general Disallow. Blockchain (2.4:1) and technology (2.3:1) have the most permissive postures. Media (0.6:1) and legal (0.7:1) are the most restrictive. The correlation between Allow-to-Disallow ratio and AI citation rate is strongly positive across industries, suggesting that clearly signaling permissive intent is rewarded by AI retrieval systems, not just the absence of blocks.
How Presenc AI Helps
Presenc AI detects and parses llms.txt for every domain we monitor, tracking adoption status, directive changes, and the correlation between llms.txt posture and measured AI citation rate on each platform. For brands considering their first llms.txt, Presenc generates an optimized starter file based on your content map and AI visibility priorities, then monitors how each AI platform actually respects your directives in practice.