llms.txt Explained: What It Is and How to Set It Up Correctly
A single text file in your root directory can decide whether AI models understand your website at all. Here's where llms.txt comes from, how it's structured, and how to create your own in 10 minutes.
What is llms.txt?
llms.txt is a markdown file in a website's root directory - reachable at yourdomain.com/llms.txt, similar to robots.txt or sitemap.xml. The difference: robots.txt tells crawlers what they're NOT allowed to visit. llms.txt tells AI models WHAT your website is and WHERE the most important content lives - a structured summary instead of an HTML page full of navigation, ads, and JavaScript.
The standard was proposed in September 2024 by Jeremy Howard and the team at Answer.AI - in response to a concrete problem: AI models' context windows are too small for entire websites, and regular HTML pages are unnecessarily hard for language models to process.
Why llms.txt matters for GEO
Many AI crawlers read llms.txt as their first point of orientation before crawling the rest of a website - similar to how Googlebot checks robots.txt and sitemap.xml first. A well-maintained llms.txt increases the odds that an AI model understands your product definition correctly, instead of guessing at it from fragmented HTML snippets.
Important context: llms.txt is not an official standard from Google, OpenAI, or Anthropic, and it doesn't guarantee citations. It's a voluntary, open proposal - but one that more and more documentation platforms and tools are adopting because it makes structural sense. It's just one of several GEO signals - the full overview is in our article on What is GEO?, along with Schema Markup for AI Citations.
How is an llms.txt structured?
# Project nameAn H1 with the name of your website or product - required by the specification.> Short descriptionA blockquote paragraph: what the website is, in one or two sentences, with no marketing fluff.## Sections with linksFreely chosen H2 headings (e.g. "Docs", "Pricing", "API") with markdown links to the most important subpages.## OptionalA section for less important links that AI models can skip if needed.# MyProduct > MyProduct is a tool for X that solves Y in under 60 seconds. ## Docs - [Getting started](https://example.com/docs/start): Quick start guide - [API reference](https://example.com/docs/api): Full API documentation ## Pricing - [Pricing & plans](https://example.com/pricing): All plans at a glance
Full specification: llmstxt.org ↗
llms.txt vs. llms-full.txt: what's the difference?
llms.txt stays deliberately short - an overview with links. llms-full.txt (optional, at /llms-full.txt) covers the same topics in more depth: detailed pricing, FAQs, technical specs. For AI models that can load more context at once, the expanded version delivers more direct, citable answers - without making users click through several linked subpages first.
Common mistakes when setting it up
Marketing copy instead of facts
llms.txt should be precise and factual - "X is a Y for Z that does A" instead of marketing language. AI models extract direct statements from it.
File blocked or behind a login
llms.txt needs to be publicly accessible without authentication - otherwise AI crawlers can't read it at all.
Never updated
New features, changed pricing, or renamed products go stale in the file if it isn't maintained alongside those changes.
Wrong format
llms.txt has to be valid markdown, not an HTML page with a .txt extension - otherwise many parsers can't process it cleanly.
Frequently asked questions about llms.txt
What is llms.txt?
llms.txt is a markdown text file in a website's root directory (similar to robots.txt) that explains, in a structured, compact form, what a website is, what it offers, and where the relevant content lives - specifically for AI models to read. It was proposed as an open standard in September 2024 by Jeremy Howard (Answer.AI).
Is llms.txt required for AI visibility?
No, it's not an official Google or OpenAI standard, and it's not required. It's a voluntary, community-driven proposal - but one that more and more websites and documentation platforms are adopting, because it gives AI crawlers clear first orientation instead of making them navigate through menus, ads, and JavaScript.
What is the difference between llms.txt and llms-full.txt?
llms.txt is a short overview - product description, key points, links to further resources. llms-full.txt (optional) contains the same information in expanded form, often including pricing, FAQs, and technical details, for AI models that can load more context.
Where does llms.txt need to live, and how do I find it?
In the domain's root directory, reachable at yourdomain.com/llms.txt - just like robots.txt or sitemap.xml. It has to be publicly accessible without a login so AI crawlers can read it.
Do you already have an llms.txt?
AuditAI automatically checks whether llms.txt and llms-full.txt exist and are correctly formatted - as part of 19 GEO signals in under 60 seconds. Start without registration, sign up free for the full report with all scores.
Check your GEO score nowWhat is GEO? How to Get Recommended by ChatGPT and Claude
llms.txt is one of 19 GEO signals - the full overview of what else counts.