Suddenly, everyone's building a "markdown version" of their site for AI agents. Developers are shipping llms.txt files, spinning up parallel content trees, and telling clients this is where AI SEO is headed. Then John Mueller from Google jumped in and called it what it actually is: technical debt wearing a strategy costume.
His comment came after someone pointed out an uncomfortable truth on Bluesky. Sites are racing to build text versions for AI agents while skipping basic accessibility work that helps actual people, things like proper heading structure and screen reader support. Mueller's response was blunt. A properly built website already works for AI agents, search engines, and humans. If you're creating a separate "agent-friendly" copy instead of fixing the real site, you're just adding work you'll have to redo later.
This matters because the rush to build markdown mirrors is repeating old SEO mistakes. Remember link directories, keyword stuffing, doorway pages? Each one promised a shortcut. Each one turned into cleanup work a year or two later. The current wave of AI SEO and Markdown enthusiasm looks a lot like those older cycles. Real agent-friendly architecture isn't about splitting HTML from Markdown. It's about separating discovery from usage, and figuring out which one actually needs a format change.
- The Markdown Myth: Why AI SEO Isn't About Duplicate Files
- Discovery vs. Usage: Where Markdown, AEO, and llms.txt Actually Fit
The Markdown Myth: Why AI SEO Isn't About Duplicate Files
A belief has taken hold that markdown files carry some secret ranking power for AI search. They don't. It's worth saying plainly before more teams sink time into building duplicate content trees for nothing.
What Search Engines Crawl
Googlebot and Bingbot have crawled HTML for over thirty years now. That hasn't changed. HTML is still the primary format that crawlers use for indexing and ranking. A markdown page sitting next to your HTML page gives you zero advantage, not in traditional SEO, and not in GEO, the practice of shaping content for generative engines like AI Overviews and chat-based search tools.
Think about it this way: if a markdown copy actually moved rankings, every SEO tool on the market would already have it on a checklist, and Google would have said so directly. Instead, Mueller's advice points the other way. Fix the actual site. Make the HTML clean, accessible, and well structured. One version, built right, can serve every visitor, human or machine.
OpenAI's own documentation backs this up. Their OAI-SearchBot crawler is described as crawling websites for search, full stop. No mention of a markdown preference, no recommendation to spin up separate text files. Sure, the bot can read a markdown file if you point it there directly, but that's not its default job. It's built to crawl your website the same way it exists for everyone else, in HTML.
The Cloaking Risk Nobody Talks About
Here's where it gets risky. Build a separate "agent version" of your content that differs from what humans see, and you're edging toward cloaking. Cloaking means showing different content to crawlers than to regular visitors, and it's been a guideline violation for decades.
Most people building markdown mirrors aren't trying to deceive anyone. They think they're helping. But intent doesn't change the technical reality. If your markdown file says one thing and your HTML page says another, even by accidental content drift, you've created the exact kind of mismatch guidelines warn against.
And this is the deeper problem Mueller was pointing at. A parallel agent version doesn't save work, it doubles it. Update pricing, a product spec, a headline, and now you need to update it in two places. Miss one, and you've got inconsistent content confusing both users and machines. That's technical debt, not a shortcut. It's the same trap as the duplicate-page schemes from early SEO history, just wearing a new name.
Discovery vs. Usage: Where Markdown, AEO, and llms.txt Actually Fit
None of this means markdown is useless. It means markdown has a job, and ranking pages isn't it. The real split is between discovery, how your content gets found and indexed, and usage, how an AI agent actually consumes your content once it's already decided to use it.
Discovery stays index-based and HTML-driven. That hasn't shifted with the rise of AI search. Usage is where a format choice like Markdown genuinely matters, particularly for answer engine optimization, or AEO, which is about getting your content correctly summarized and cited inside AI-generated answers.
Token Efficiency for Developer Docs
The clearest legitimate case for markdown is developer documentation. When a coding assistant or agent reads your API docs, markdown is lighter than HTML. It strips out nav menus, ads, scripts, and styling, leaving just the content behind. Fewer tokens spent parsing junk, which adds up fast when an agent is chewing through hundreds of pages of technical reference material.
That's why Cloudflare built infrastructure that automatically generates markdown versions of pages, aimed mostly at cutting token consumption for coding tools. It's a real, practical benefit. But notice the context: it's for tool consumption inside developer workflows, not for helping your marketing page show up in AI Overviews.
Content Negotiation Done Right
If markdown has a place, the smart way to offer it is through content negotiation, not a pile of duplicate static files. HTTP has supported this for years via the Accept header, defined in RFC 7231. A client requests a resource, specifies Accept: text/markdown, and your server responds with the markdown version of that same content, generated from a single source.
This sidesteps the drift problem entirely. One canonical version of your content. Change it once, and every format served from it updates automatically. No stale copies, no mismatched pricing, no cloaking risk, because you're serving different formats of identical content based on a standard request header, not hiding information from certain visitors.
