llms.txt
llms.txt is a proposed markdown file served at the root of a website — /llms.txt — that gives LLM-based tools a curated, condensed map of a site's most important content. Proposed by Jeremy Howard in 2024, it aims to do for AI what sitemap.xml did for search: make the best parts of your site discoverable and digestible at machine speed.
llms.txt is a proposed markdown file served at the root of a website — /llms.txt — that gives LLM-based tools a curated, condensed map of a site's most important content. Proposed by Jeremy Howard in 2024, it aims to do for AI what sitemap.xml did for search: make the best parts of your site discoverable and digestible at machine speed.
Why It Matters
LLMs reading the web face a context-window problem: a single marketing site can blow past 200K tokens of HTML, CSS, and navigation boilerplate before the model reaches the actual content. llms.txt solves this by providing a short, curated list of the pages the site owner wants an LLM to read, written in clean markdown without cruft. Anthropic, Cloudflare, Mintlify, Zapier, and Stripe all published llms.txt files in 2024, drawing real attention. But as covered below, no major platform — including Google and OpenAI — has adopted the file as of 2026, so treat it as a near-zero-cost optional experiment, not a proven channel.
What It Looks Like
A basic file:
# inblog
> inblog is an AI-powered blogging platform for SEO-optimized content.
## Docs
- [Getting started](https://inblog.ai/ko/docs/start-inblog/getting-started-with-inblog): Create your first blog
- [API docs](https://inblog.ai/ko/api-docs): Publish posts via API
## Optional
- [Blog](https://inblog.ai/blog): SEO and GEO insights
- [Glossary](https://inblog.ai/glossary): SEO/GEO/marketing glossary
Two sections: a heading + summary, then curated links grouped by purpose. The Optional section lists content an LLM should read only if depth is needed.
llms.txt vs robots.txt vs sitemap.xml
| File | Audience | Purpose |
|---|---|---|
robots.txt | Crawlers | What not to crawl |
sitemap.xml | Search engines | Complete list of pages to index |
llms.txt | LLM-based tools | Curated, prioritized content for ingestion |
robots.txt is a fence. sitemap.xml is a phone book. llms.txt is a curator's recommendation shelf. They're complementary, not replacements.
Two Variants
llms.txt: The short curated map — the table of contents.
llms-full.txt: An expanded version where each linked page's markdown content is inlined, giving an LLM the entire ingestible corpus in one file. Used by doc sites like Anthropic's and Mintlify's clients.
How to Author a Good llms.txt
1. Lead with a one-line positioning statement: The blockquote after the H1. This is what the LLM learns about your brand's identity.
2. Group by purpose, not structure: "Docs," "Guides," "API Reference," "Case Studies" — not "Category A," "Category B."
3. Write link descriptions as facts, not marketing: "Built-in SEO optimization" beats "Supercharge your content."
4. Put the most important pages first: LLMs under context pressure read top-to-bottom.
5. Use Optional for deep-cut content: Things the LLM should skip unless the user wants detail.
6. Update it when the site changes: An outdated llms.txt is worse than none.
Limitations — No Major Platform Has Adopted It
The single most important fact before adopting llms.txt: as of 2026, no major AI platform officially supports the file.
Google does not support it: Google's Gary Illyes confirmed at Search Central Live in July 2025 that Google doesn't support llms.txt and isn't planning to. John Mueller compared it to the (ignored) keywords meta tag, and Google's generative-AI search guide, updated in May 2026, explicitly states you don't need separate text files for AI. Google folks point to the W3C draft standard WebMCP as the direction they prefer.
Neither does OpenAI: OpenAI's crawler documentation covers robots.txt only — llms.txt appears nowhere.
Adoption has stalled: SE Ranking's study of 300,000 domains puts adoption around 10% and flat, and one 90-day measurement found only ~408 llms.txt requests out of 500M+ AI bot visits.
Not a ranking signal: It affects ingestion quality in some LLM tools (developer tools like Cursor), not SERP position.
Requires discipline: A stale llms.txt misleads the very models you're trying to reach — and it can't fix weak content.
Why inblog Sites Should Consider It
Every blog on inblog is a content surface that AI tools might ingest. Don't expect much from it — major search and AI platforms ignore the file — but a small llms.txt at the blog root costs almost nothing, and some developer tools and agents do read it. The real levers on AI citation quality remain content quality, structure, and brand mentions; llms.txt is a lightweight add-on, not a strategy.
Sources: