Common Crawl read 584,107 llms.txt files and most were plugin output

Common Crawl analysed 584,107 llms.txt files taken from its own July 2026 crawl archive and found that 68 percent of them were produced by a plugin or a template rather than written by hand, that 22 percent contained no links at all, and that the format carries no mechanism to enforce anything written inside it. Search Engine Journal reported the analysis on 8 September 2026, citing Common Crawl's own framing that the file "grants nothing and blocks nothing, and no crawler is obliged to read it." If llms.txt is on your invoice as an AI SEO deliverable, this is the largest published sample so far of what those files actually contain.

What Common Crawl measured, and on how many files

Common Crawl ran the analysis across 584,107 llms.txt files found in its July 2026 crawl archive, which is the single largest published count of the format to date. Search Engine Journal reported on 8 September 2026 that 49 percent of the files had the full structure the format describes, meaning an H1 title, a summary blockquote and link sections, and that 32 percent included short descriptions alongside the links. Structural compliance, in other words, is not the problem. Roughly half the files are shaped correctly. The problem sits in what the correctly shaped files say and who wrote them.

The headline finding is authorship. Common Crawl found that 68 percent of the 584,107 files originated from a plugin or a template. That leaves under a third of the sample as anything a person sat down and thought about. Alongside that, 22 percent of the files contained no links at all, which for a format whose entire purpose is to point a language model at the pages worth reading is closer to an empty envelope than a document.

A smaller slice tries to do something the format cannot do. Common Crawl found that 6 percent of the files included guidelines on how AI systems should use the site, such as rate limits or copyright notices. Those lines are text in a file that no crawler has agreed to obey. They read like rules and function like a comment.

Three generators account for most of what the format looks like in the wild

Common Crawl's breakdown by generator explains the 68 percent better than any argument about best practice. Wix alone accounted for 41 percent of the 584,107 files analysed, so nearly half of everything measured is one platform's default output. When a single site builder produces that share of a format's population, the format's observable characteristics are that builder's product decisions rather than a community norm.

All in One SEO's implementation goes the other way. Common Crawl reported a median of 138 links per file across roughly 73,000 files from that plugin, and described the tool as treating llms.txt as a sitemap. A 138-link median is not a curated reading list. It is an index, dumped in a second place, in a format with no crawler contract behind it.

The third pattern is the one worth remembering. Common Crawl found that GoDaddy's parked-domain files pass every structural check without containing links, because the file is a sales pitch. A validator that only checks for an H1, a blockquote and section headings marks those files as correct. Structure, on its own, tells you almost nothing about whether a file is useful.

The blocking that was not blocking

The most testable finding in the analysis concerns crawler directives. Common Crawl found that 1,570 of the files named specific crawlers, and that 32 of them appeared to deny CCBot, Common Crawl's own crawler. It then went and read those sites' real robots.txt files. Of the 31 sites it could assess, not one blocked CCBot outright. Five explicitly allowed it, 11 limited it to certain paths, and 15 had no restrictions on it at all.

That gap is the whole argument in one number. The sites had written a denial into a file with no enforcement path and left the file that does have one open. Search Engine Journal reported the conclusion plainly: a crawler restriction written into llms.txt does nothing by itself, and a rule has to appear in robots.txt to be enforced.

Common Crawl also found 136,578 robots.txt files sitting at the llms.txt path, which suggests a large number of sites answer any request to that path with whatever their server has lying around. If you have never opened your own file, that is a plausible thing to find there.

Prompt injection turned out to be a smaller story than the warnings

The security angle that circulated when llms.txt first appeared did not survive the sample. Common Crawl found that 10 files matched its strictest prompt injection test and that only four were genuine cases, all of them inserted intentionally by people making a point rather than attacking anyone. Four deliberate demonstrations across 584,107 files is not a security problem.

Milder steering was more common but still marginal. Common Crawl counted 3,793 files containing softer guidance along the lines of asking a model to focus on particular pages. That is roughly six files in a thousand, and a request a model is free to ignore is not an exploit.

How this differs from what we published in August

Relevant Audience has covered this thread twice already, and this study is a different object. On 10 August 2026 we published "A file about office cats broke the llms.txt evidence," which was about a single test case and what it showed about the evidence base for the format. On 11 August 2026 we published "Common Crawl published an AI visibility audit, and the worst blocks never returned a page," which was about crawler access and blocking behaviour.

The 8 September study is neither. It is a content census: a very large sample examined for what the files say, who generated them, whether they contain links, and whether their stated crawler rules match the site's real robots.txt. The earlier pieces asked whether the file is reachable and whether the evidence for it holds up. This one asks what is inside the files that exist, and the answer is that most of them were written by software.

The numbers, in one place

Every figure below comes from Common Crawl's analysis of its July 2026 crawl archive as reported by Search Engine Journal on 8 September 2026.

What the study explicitly did not say

Common Crawl stated its own limitation: the sample is random only within sites accessible to the crawler, not the entire web. That matters for anyone comparing this study with other llms.txt adoption counts. A site that blocks CCBot, sits behind a login, or never got crawled is not in the 584,107, so the adoption figures here are not directly comparable with numbers published elsewhere.

Three other things are absent from the reporting and should not be filled in with guesswork. The study was not broken out by country or by language, so there is no Thai figure and no way to tell from this data whether Thai sites behave differently from the sample as a whole. The reporting does not establish whether any AI company reads llms.txt at all, only that no crawler is obliged to. And it does not say that the format is finished or that anyone should delete their file. It says the deployed population is mostly automatic output.

A self-audit you can finish in five minutes

A sample of 584,107 files is useful mainly because it tells you what your own file probably looks like before you open it. Open yourdomain.com/llms.txt in a browser and work down this list.

• Does the file load at all, or does your server hand back something else? Common Crawl found 136,578 robots.txt files served at the llms.txt path, so getting the wrong document back is a documented outcome, not a strange one.
• Does it contain links? Twenty-two percent of the sample did not. A file with no links cannot do the one job the format was proposed for.
• Did a plugin write it? If the link list runs to dozens or hundreds of entries and matches your sitemap, you are looking at the All in One SEO pattern, where the median was 138 links per file. That is a sitemap with a different filename.
• Does it contain crawler directives, rate limits or copyright language? Six percent of the sample did. Those lines belong in robots.txt if you want them enforced, and the 31 assessable sites that denied CCBot in llms.txt while allowing it in robots.txt show what happens when they stay in the wrong file.
• Does your robots.txt say the same thing your llms.txt says? If the two disagree, robots.txt is the one crawlers act on.

None of that requires a tool or a subscription. If the file turns out to be plugin output with no curation behind it, the honest description is that you have a file, not that you have done AI SEO work.

What this means for Thai marketers

Wix and All in One SEO are both common on Thai SME sites, and both are named in Common Crawl's generator breakdown, so the patterns described here are likely to show up locally. That is an inference about platform usage, not a finding: the study was not broken out by country or language, and there is no Thailand figure in it.

The practical consequence is about what gets sold. llms.txt has become one of the easiest line items to put on an AI SEO proposal because it is a small text file that can be generated in seconds, and this study is the largest evidence yet that most deployed files are exactly that. If a proposal lists llms.txt as a deliverable, the reasonable question is what the file will contain, who curates the link list, and what it does that the sitemap does not. If a supplier says it will stop AI crawlers from taking content, that claim is contradicted by the 31 assessable sites Common Crawl checked.

The work that does move AI visibility is the work that survives whether or not a crawler reads a text file: pages that answer a question in the first sentence, facts stated with the actor and date inside the sentence, tables and question-and-answer blocks a model can lift cleanly, and content that other sites cite. That is what generative engine optimisation is actually made of, and none of it depends on a file that grants nothing and blocks nothing.

Frequently asked questions

Should I delete my llms.txt file?

The study does not say to delete it, and neither do we. Common Crawl reported that the file grants nothing and blocks nothing, which is an argument against relying on it, not an argument for removing it. A hand-curated file that points at your genuinely useful pages costs nothing to keep. A plugin-generated duplicate of your sitemap is not worth defending either way, and the decision matters far less than what is on the pages it points to.

Does this apply to Thai websites?

The source did not break the data out by country or language, so there is no Thailand-specific figure in it. What can be said is that Wix accounted for 41 percent of the files analysed and All in One SEO for roughly 73,000 of them, and both are widely used in Thailand, so the same generator patterns are likely present here. Anything more specific would be a guess.

Can I use llms.txt to block AI crawlers from my content?

No, and the study measured exactly this. Common Crawl found 32 files that appeared to deny its CCBot crawler, checked the 31 sites it could assess, and found that none of them blocked CCBot in robots.txt: five explicitly allowed it, 11 limited it to certain paths and 15 had no restrictions. A directive has to be in robots.txt to be enforced.

Was llms.txt being used to attack AI models with prompt injection?

Barely. Common Crawl found 10 files matching its strictest prompt injection test across 584,107 files and judged only four to be genuine, all of them planted deliberately by people demonstrating the idea. A further 3,793 files contained milder guidance such as asking a model to focus on certain pages.

Do I have to do anything today?

Open your own file and check whether it has links and whether a plugin wrote it. That is the whole action. If your file contains crawler rules, move them to robots.txt, which is the file crawlers act on. Nothing in this study requires a rebuild, a migration or new software.

Where to take this next

A 584,107-file census is useful mostly as a calibration exercise. It tells you that the deliverable is cheap, that the enforcement everyone assumed was there is not, and that the file most sites are running was written by their CMS. If you want to know what your own site looks like to an AI answer engine, the file is a five-minute check and the pages are the actual work. If you would like a read on which of your pages are structured to be quoted rather than merely crawled, that is a conversation worth having.