An llms.txt file is a plain text file you put at the root of your website. It is meant to tell AI assistants what your business does and which pages matter, in a format they can read quickly. The idea has been floating around since 2024 and nobody agreed whether it caught on.
So we counted. In August 2026 we took 589 Australian websites that had surfaced in real customer searches across solar, legal, building, trades, dental, finance, health and removals. Then we checked every one of them for an llms.txt file, and read their robots.txt to see whether they block AI crawlers.
181 of the 589 sites had an llms.txt file. That is 31%.
We were not expecting that. Then we read them, and the picture changed.
Key findings
- 31% of the Australian sites we checked publish an llms.txt file.
- A third of those were generated automatically by an SEO plugin, not written by anyone. Yoast, All in One SEO and Rank Math accounted for 59 of them.
- 32% of llms.txt files contain none of the six facts an AI needs to recommend a business. They are lists of links.
- Only 10% of sites block any AI crawler in robots.txt, and the ones they block are mostly training crawlers, not search crawlers.
- 13% of sites served a normal browser fine but returned an error to an AI crawler. This is the real blocking problem, and almost nobody knows it is happening.
How we tested
The 589 domains came from live Australian searches, so they are real businesses competing for real customers, not a curated list. For each one we requested the homepage twice, once as a normal Chrome browser and once as GPTBot. We requested /robots.txt, /llms.txt and /.well-known/llms.txt. We then re-fetched every llms.txt we found and read what was inside it.
Finding 1: llms.txt adoption is real, but mostly accidental
181 sites had one. Then we looked at who wrote them.
| Who generated the file | Files | Share |
|---|---|---|
| Yoast SEO | 34 | 19% |
| All in One SEO | 13 | 7% |
| Rank Math | 12 | 7% |
| No generator named | 122 | 67% |
At least a third were switched on by a plugin update. The business owner did not decide to publish one. Their SEO plugin shipped the feature and turned it on.
That matters because of what those files contain.
Finding 2: most llms.txt files say nothing useful
When an AI recommends a business, it quotes specifics. Years in business. A licence number. What a price includes. Which suburbs you cover. We checked every llms.txt file for six of those facts.
| Fact | Share of llms.txt files that include it |
|---|---|
| Areas or suburbs served | 44% |
| A price signal | 34% |
| Year founded or years in business | 27% |
| Response time or availability | 25% |
| Licence, ABN or accreditation number | 17% |
| Review count or star rating | 10% |
32% contained none of the six. Only 13% contained four or more.
The median file was 7,003 bytes and listed an average of 119 links. That is the shape of an auto-generated file: a sitemap in a different coat. It tells an AI where your pages are. It does not tell it why you should be recommended.
Twenty-four sites out of 589 had an llms.txt that was hand-built and carried four or more of the six facts. Four percent. That is the real adoption number.
Finding 3: hardly anyone blocks AI crawlers on purpose
The scary version of this story is that businesses are locking AI out. We checked, and they are not.
488 sites had a readable robots.txt. 107 of them (22%) named an AI crawler at all. Only 47 (10%) fully blocked one.
| Crawler | What it does | Sites blocking it |
|---|---|---|
| Bytespider | TikTok training | 9% |
| CCBot | Common Crawl training | 9% |
| GPTBot | OpenAI training | 8% |
| ClaudeBot | Anthropic training | 7% |
| meta-externalagent | Meta training | 7% |
| Google-Extended | Google AI training | 6% |
| PerplexityBot | Perplexity search | 1% |
| OAI-SearchBot | ChatGPT search | 0.4% |
Look at the bottom two rows. The crawlers that actually fetch pages to answer a customer’s question are almost never blocked. The ones being blocked are training crawlers.
That is a defensible choice. Blocking GPTBot stops your content being used to train a model. It does not stop ChatGPT citing you, because ChatGPT uses a different crawler when it searches.
If you have ever been told “you blocked GPTBot, that’s why AI ignores you”, that is usually wrong.
Finding 4: the real blocker is your security, not your robots.txt
Here is the number worth acting on.
Of the 500 sites whose homepage loaded fine in a normal browser, 65 of them (13%) returned an error when the same page was requested by an AI crawler.
| What the AI crawler got back | Sites |
|---|---|
| 403 Forbidden | 48 |
| 429 Too Many Requests | 11 |
| Other errors | 6 |
None of these businesses chose this. It is a firewall or bot protection default, usually a CDN or hosting security setting, deciding that anything without a browser fingerprint is suspicious.
The effect is the same as blocking AI on purpose. Your page cannot be read, so it cannot be quoted.
This is the cheapest thing on this page to fix and the most commonly missed.
So should you publish an llms.txt?
Honest answer: it will not hurt, and we have no evidence yet that it helps on its own.
No major AI company has confirmed it uses llms.txt as a ranking or citation input. What we can say from this study is that the sites getting cited in AI answers were not the ones with the biggest llms.txt files. They were the ones with checkable facts on ordinary pages.
Our order of work, cheapest and most certain first:
- Check that an AI crawler can fetch your homepage. If it gets a 403, nothing else matters. This affects one site in eight.
- Check your robots.txt actually says what you meant. If you want to be in AI answers but not in training data, block the training crawlers and leave OAI-SearchBot and PerplexityBot alone.
- Put the six facts in plain text on your pages. This is what gets quoted. We covered it in our study of 500 Australian homepages.
- Then publish an llms.txt if you want. Write it properly. A generated link list adds nothing.
If you would rather have someone check all four on your site, that is the first thing our answer engine optimisation work does.
Common questions
What is llms.txt?
A plain text file at the root of your website, at /llms.txt, that describes your business and key pages in a format AI assistants can read easily. It is a proposed standard, not an official one.
Does llms.txt actually work?
There is no public confirmation from OpenAI, Google, Anthropic or Perplexity that they use it. In our study, having one did not line up with being cited. Treat it as low cost and unproven rather than essential.
How many Australian businesses have an llms.txt?
31% of the 589 sites we checked. But a third of those were generated automatically by an SEO plugin, and only about 4% of all sites had a hand-written file with real information in it.
Should I block GPTBot?
Only if your goal is to keep your content out of AI training data. Blocking GPTBot does not remove you from ChatGPT’s answers, because ChatGPT uses OAI-SearchBot to fetch pages when it searches. Blocking that one would remove you.
Why can’t AI find my website?
The most common reason we found is not robots.txt. It is that your firewall or bot protection returns a 403 to anything that does not look like a browser. This affected 13% of the sites we tested and none of them appeared to have chosen it.
How do I check if AI can read my site?
Request your own homepage with an AI crawler user agent and see what comes back. If you get a 403 or 429 while a browser gets a 200, your security settings are blocking AI. Your host or whoever manages your firewall can allow the AI crawlers by name.
Is llms.txt the same as robots.txt?
No. Robots.txt tells crawlers what they may not fetch and is universally respected. Llms.txt tries to tell AI what your business is and is not officially supported by anyone yet.
Method: 589 Australian domains collected from live commercial searches were each requested in August 2026 for /robots.txt, /llms.txt, /.well-known/llms.txt and the homepage, using both a Chrome user agent and a GPTBot user agent. Robots.txt was parsed for full disallow rules against 18 named AI crawlers. Every llms.txt found was re-fetched and checked for six categories of business fact. Figures reflect that date.
