llms.txt is a simple markdown file that you place in your website's root directory and that gives AI systems a curated map of your most important pages. The name is self-explanatory: "llms" stands for large language models. The idea is as simple as a table of contents: instead of letting an AI system crawl your site blindly, you show it which pages it should know about and what each one is about.
In this article we explain what llms.txt is, and what it is not, and how it relates to robots.txt and your sitemap, closing with a short checklist to decide whether the file makes sense for you.
What exactly is llms.txt?
The proposal comes from the open web community and is deliberately simple: llms.txt lives in the root of your domain (that is, at https://yourdomain.com/llms.txt), is a markdown file, and follows a basic pattern: a title, a short description of your company or project, then a list of links with one-line explanations of why each page matters. A longer llms-full.txt can optionally carry the full text for systems that want the details.
Think of the file as a reception desk: instead of telling the visitor (that is, the AI system) to "look around the building yourself", a friendly member of staff says, "the important rooms are here." That is exactly what llms.txt does: it points the AI system towards what you want it to see first.
What can llms.txt do and what can it not?
An honest assessment matters, because many articles promise more than the file can deliver:
- No ranking guarantee: llms.txt does not automatically mean that ChatGPT, Perplexity or Google will mention your website more often. As with everything in AI visibility, the file helps, but it does not replace good content.
- No official standard yet: no major search engine has made llms.txt a requirement so far. Many AI systems still do not read it routinely, and it is impossible to say how many will in future.
- No substitute for the basics: if your website is slow, hard to navigate, or has thin content, even the best llms.txt will do little.
- A cheap, easy extra: the file costs you almost nothing, a few sentences you already know, and can make your key pages easier for AI systems to find.
How does llms.txt relate to robots.txt and your sitemap?
The three files do different jobs and complement rather than replace each other:
| File | What it does | Who reads it? |
|---|---|---|
robots.txt | Decides which areas crawlers may enter | Search engine crawlers, AI crawlers |
sitemap.xml | Lists every indexable URL on the site | Mainly search engines |
llms.txt | Gives AI systems a curated, readable overview | AI systems / LLMs |
robots.txt is the bouncer: it decides who gets in. The sitemap is the floor plan: it lists every room. llms.txt is the guided tour: it shows a visitor the important rooms right at the start. Keep the files consistent, if robots.txt blocks an entire section, that section should not also appear in llms.txt as "must read".
Checklist: does your website need an llms.txt?
- Your website has a clear, sensible structure
- You know which 5 to 15 pages actually matter to your customers
- You can write a short one-line description for each of those pages
- You want AI systems to find and summarise your website more easily
- You accept that the file is a bonus, not a guarantee
If most of those points fit, it is worth doing: write a short markdown file, place it in your website's root directory and check that it loads at the right address. We regularly set it up as part of our SEO and AI visibility projects, where we also check whether the rest of your site structure is already AI-friendly.
Conclusion: llms.txt is a table of contents, not a magic box
llms.txt makes your website clear, organised and quick for AI systems to take in, which it should be anyway. It is a proposal, not a standard, and it promises nothing, but it costs you almost nothing and fits well with a website that explains its key content clearly to people, too.
If you want to find out where your website stands with AI systems, get in touch. We will look at it together and work out what makes sense for your site.
