Shared Claude conversations showed up in Google search results over the weekend, findable with the site: operator, and 404 Media reported it on the Monday. Search Engine Journal checked claude.ai's crawl rules on 27 July and found a conflict in how those pages are controlled: the share path is blocked in robots.txt, while the pages themselves serve a header telling Google not to index them. Running both controls on the same URL does not double the protection. It cancels it.

The lesson travels well beyond one chatbot. If any part of your site stays out of Google on the strength of a single Disallow line, the same hole is in your setup.

What the crawl rules said

SEJ documented two directives pointing in opposite directions.

robots.txt disallows /share/* under the User-agent: * group. There is no separate Googlebot group, so that is the group Google follows.
A live share URL returns the header x-robots-tag: none on a GET request. Google's robots meta guidelines treat none as equivalent to noindex plus nofollow.

SEJ saw the same header when the request claimed to be Googlebot, and the response carried Vary: User-Agent. Each control looks reasonable on its own. On the same URL, they stop working.

Why a Disallow plus a noindex cancel out

A noindex only works if the crawler is allowed to fetch the page and read it. The directive lives either in the HTTP response header or in the page HTML, and both of those require a request that robots.txt permits. Block the path and Googlebot never opens the file, so it never sees the instruction not to index.

Meanwhile a blocked URL can still end up in the index. When other pages link to it, Google can note the URL without opening it and list it on the strength of those links. Blocking the crawl is not the same as blocking the listing, and this is the part that catches people out.

Google has been saying this for years. John Mueller has said pages blocked by robots.txt can still appear in results if other pages link to them. Martin Splitt recommends against putting both rules on the same page. For removal, the guidance has been consistent: serve a noindex the crawler can actually read.

What is not established

SEJ could not establish what was served to Google when the URLs were first discovered, or whether that header was in place before the weekend. The conflict was verified on 27 July. The indexing happened earlier. Independent IT consultant Daniel J. Glover reported the same conflict on 26 July. Anything beyond that about what Google was shown at the moment of indexing is speculation, and should be treated as such.

The /public/artifacts/ path is not listed in robots.txt at all. SEJ did not verify whether artifact URLs return a page-level noindex, so the two paths cannot be compared on their controls. It is not accurate to say one is protected and the other is not.

On what surfaced: TechCrunch reported finding medical material and internal company documents among the pages. Anthropic told TechCrunch that share links appear in search results only when people post them somewhere crawlers can reach, and that links sent through private messages are not indexed. Users can stop it at their end by unsharing chats in Settings > Privacy > Shared Chats.

Where the same hole usually sits on a commercial site

The pattern is a public, linkable URL directory kept out of search by one line in robots.txt:

Share links and public report links generated by a tool or dashboard
Client portals and quote pages that are unlisted rather than locked
Staging and preview subdirectories on the live domain

None of that is a problem while nobody links to the URLs. The moment one gets pasted into a forum, a social post or a partner site, the Disallow stops being a defence. This is the sort of thing a structured technical SEO audit is supposed to surface before it becomes a search result.

The fix, in order

Decide whether the URL should be publicly reachable at all. If the content is confidential, robots directives are the wrong tool in either configuration. Put it behind authentication.
If it stays public but should not be listed, allow the crawl and serve the noindex. Remove the Disallow line for that path and keep x-robots-tag: noindex or the meta robots tag. Let Google fetch the page, read the directive, and drop it.
Keep a Disallow only where you can live with the URL being listed. What it buys is crawl savings, not invisibility.
Re-check after recrawl. Removal is not instant: Google has to fetch the page again to see the directive it was blocked from reading.

How to check your own site

Run a site: query with the directory appended and see what is listed.
Open a sample URL in the URL Inspection tool in Search Console and read its crawl and index status.
Request the headers with curl -I and confirm whether an x-robots-tag is present.
Open your robots.txt and, for every Disallow line, ask what happens if a public page links to a URL in that path.

What this means for marketers in Thailand

Most Thai sites that carry this risk carry it in the same two places: a staging subdirectory left on the production domain during a rebuild, and a link-sharing feature blocked in robots.txt because it was the quickest fix at the time. Both are common on sites with a long agency and developer history, where nobody owns the robots.txt file any more.

Bilingual sites add a second version of the problem. A Thai directory blocked while the English equivalent is not, or a language switcher pointing at a disallowed path, produces gaps that are hard to read from the index alone. Treat robots.txt as a file with an owner and a review date, not something touched once at launch, and keep it in the same maintenance rhythm as the rest of your SEO programme.

FAQ

Does a Disallow keep a page out of Google? It keeps Googlebot from fetching it. It does not guarantee the URL stays out of results, because Google can list a URL it has never opened when other pages link to it.

Which directive removes a page from search? A noindex the crawler is allowed to read, which means no Disallow on the same path.

If you are not sure which of your directories are blocked and which are indexed, close that gap before someone finds it for you. Our team can walk your robots.txt and index coverage with you.