TL;DR
- Researchers from NUS and Xi'an Jiaotong University posted an arXiv preprint on 11 October 2026 tracing 1,600 runs each of GPT-5.3 Instant and GPT-5.5 Instant across 80 questions.
- Google results plus OpenAI's Responses API recovered only 28.3% (GPT-5.3) and 26.2% (GPT-5.5) of the URLs ChatGPT cited, but 50.3% and 48.7% of cited hostnames.
- Google's top five pages of organic results alone contained just 9.4% and 6.4% of the exact URLs ChatGPT cited.
- No observed GPT-5.3 or GPT-5.5 Instant run searched with the user's exact wording, and 67.0% to 69.6% of Chinese-question runs searched in English only.
- Across 20 repeat runs, 40.8% to 44.5% of retrieved URLs appeared only once; the study did not test Thai prompts or any commercial tracking tool.
Google search results and OpenAI's own Responses API do a poor job of reproducing the exact pages ChatGPT cites. In an arXiv preprint posted on 11 October 2026, researchers from the National University of Singapore and Xi'an Jiaotong University found that the two proxies combined recovered only 28.3% of the URLs cited by GPT-5.3 Instant and 26.2% of those cited by GPT-5.5 Instant. At the level of whole sites, the match nearly doubled, to 50.3% and 48.7% of cited hostnames.
The paper is titled "Understanding ChatGPT Citations: Evidence from Production Web Search Across Model Generations" and is by Yunxuan Jiang, Shiqi Zhang, Anqi Xing, Xiangyu Chang and Xiaokui Xiao. It was published on arXiv as a preprint, which means it has not been through peer review. Search Engine Journal wrote it up on 18 October. Every figure below comes from the paper itself.
What the researchers measured
Most studies of AI search look at the final answer: which links appear under it. This one looked at what happens before the answer. The authors recorded the server-sent event (SSE) stream that ChatGPT's web interface returns to a signed-in session. That stream exposes the searches ChatGPT actually ran, the pages those searches returned, and the pages it picked as references.
They did this for four ChatGPT web settings: GPT-5.2 Instant, GPT-5.3 Instant, GPT-5.4 Thinking and GPT-5.5 Instant. Collection ran between 29 March and 29 July 2026. The questions came from Search Arena, a public crowd-sourced dataset of 24,000 real interactions with search-augmented chatbots. The team sampled 80 questions across four intent types (factual lookup, information synthesis, analysis and recommendation), with 20 per intent, split evenly between English and Chinese.
GPT-5.3 Instant and GPT-5.5 Instant each answered all 80 questions 20 times, for 1,600 runs per model. GPT-5.2 Instant (547 runs over 32 questions) and GPT-5.4 Thinking (491 runs over 30 questions) have thinner coverage because OpenAI deprecated them before collection finished. The headline comparisons in the paper therefore use the two complete settings, 5.3 and 5.5.
Against those runs, the researchers built two outside stand-ins, the kind of data a monitoring tool can get without watching a real ChatGPT session:
- Google Search: up to five pages of organic results per question, pulled through SerpAPI. That averaged 43.2 results per question.
- OpenAI Responses API: gpt-5.3-chat-latest and gpt-5.5 called with web search on and no system prompt, 20 times per question each, so 1,600 calls per model. An average response listed about 26 to 28 candidate sources and 5 to 6 citations.
Both proxies were collected within the same three-day window as the matching ChatGPT runs.
The result: close on sites, far off on exact pages
The core finding is that outside data finds the right sites far more often than the right pages. The table below gives the paper's coverage (recall) figures from its Table 4: the share of pages, or hostnames, cited by ChatGPT that each proxy also contained.
| Proxy (share of ChatGPT-cited items it contained) | GPT-5.3 Instant | GPT-5.5 Instant |
|---|---|---|
| Google organic results, exact URLs | 9.4% | 6.4% |
| Google organic results, hostnames | 26.1% | 22.8% |
| Responses API full source list, exact URLs | 22.9% | 22.5% |
| Responses API full source list, hostnames | 38.3% | 38.1% |
| Google plus API combined, exact URLs | 28.3% | 26.2% |
| Google plus API combined, hostnames | 50.3% | 48.7% |
Two things stand out. First, Google's organic results on their own contained fewer than one in ten of the exact pages ChatGPT cited. A page ranking well in Google is not the same as a page ChatGPT ends up citing, at least for the questions in this sample. Second, even with both proxies stacked together, about three in four cited URLs were missing.
The API's own citations, as opposed to its full source list, behaved differently. They covered less ground but were more precise. For GPT-5.5 Instant, 23.3% of the URLs the API cited were also cited by ChatGPT, up from 16.4% for GPT-5.3 Instant. At the hostname level that precision rose from 27.7% to 40.5%. In other words, the API's citation behaviour sat closer to ChatGPT's newer model than to the older one, while the amount it covered barely moved.
The authors also tested an entity layer: named things such as brands, products, people and places pulled from page titles and snippets. The combined proxies recovered 50.7% of cited entities for GPT-5.3 Instant and 45.1% for GPT-5.5 Instant. The paper notes that brands were counted inside this layer and not analysed separately, so it does not support a brand-only conclusion.
ChatGPT does not search for what the user typed
In a step the paper calls query rewriting, ChatGPT turns the user's question into one or more search queries of its own, and those are what decide which pages come back. The paper found that the exact user wording was used as a search query in 5.5% of GPT-5.2 Instant runs and in none of the observed GPT-5.3 Instant or GPT-5.5 Instant runs.
Language changed too. For the Chinese questions, 96.8% of GPT-5.2 Instant runs issued both Chinese and English queries. For GPT-5.3 Instant and GPT-5.5 Instant, 67.0% and 69.6% of runs searched in English only.
The models also differ in how hard they search. GPT-5.3 Instant averaged 1.06 queries per run and GPT-5.5 Instant 1.45, and both were almost always single-round (multi-round rates of 5.5% and 3.8%). GPT-5.4 Thinking was a different animal: 11.82 queries over 3.20 search rounds per run on average, with 71.3% of runs searching more than once. The share of its queries using the site: operator rose from 25.1% in the first round to 54.7% in later rounds, and quoted phrases from 1.0% to 22.7%. The paper treats the Thinking data as supplementary because its question set was not matched to the Instant settings.
One run is not a measurement
Ask ChatGPT the same question twice and it often reads different pages. Across 20 runs of each question, 40.8% of the distinct URLs retrieved by GPT-5.3 Instant and 44.5% of those retrieved by GPT-5.5 Instant showed up in only one run. Only 9.4% and 7.7% were "stable core" pages that appeared in at least 14 of 20 runs. GPT-5.5 Instant pulled more distinct URLs per question (132.2 against 96.4) without becoming more consistent.
Stability improved at coarser levels. Pairwise overlap between runs (Jaccard similarity) was about 0.30 for URLs, about 0.40 for hostnames and 0.40 to 0.42 for entities. Repeated runs tend to visit different pages on the same sites.
Repetition also carried signal. Using a leave-one-run-out test, pages that appeared more often in the other 19 runs were more likely to be cited in the held-out run. For GPT-5.5 Instant that signal scored an AUC of 0.651 at URL level, 0.709 at hostname level and 0.711 at entity level. The authors stress that this is an observed association: it does not show that appearing more often causes a citation.
The study also flagged a practical trap. During collection, two sessions silently switched to fallback models after 14 and 16 successful runs on the intended model. Anyone running repeated prompts to track visibility could end up measuring a different model without noticing.
What predicts a citation, and what does not
The paper looked at which observable traits separate cited from uncited pages. Snippet length showed a weaker but consistent signal: 16.5% and 16.9% of cited snippets ran to 500 characters or more, against 1.5% and 1.9% of uncited ones. Retrieval position was the clearest of the remaining signals, with AUCs of 0.561 for GPT-5.3 and 0.623 for GPT-5.5, but pages beyond position 15 were still cited at rates of 24.0% and 19.0%. Publication recency was close to no signal at all, with AUCs of 0.482 and 0.535.
Finally, the authors combined the outside signals (rank, API exposure frequency, how often an exposed page was cited by the API, the number of supporting URLs, snippet length, intent and language) in an XGBoost model. Inside the outside candidate set, it reached AUCs of 0.782, 0.763 and 0.743 for GPT-5.3 Instant at URL, hostname and entity level, and 0.779, 0.763 and 0.742 for GPT-5.5 Instant. That is useful, but it is capped by coverage: a cited page that neither Google nor the API surfaced never enters the model. The paper states that even perfect ranking inside that set could recover only 26.2% to 28.3% of all cited URLs.
Where the study stops
The limits matter as much as the numbers, and the paper is clear about most of them:
- It is a preprint, not peer reviewed, and it covers 80 questions in English and Chinese. It did not test Thai or any other language.
- Its definition of "cited" includes supporting pages attached to a selected reference, even when those pages are not shown to the user as separate links.
- It covers ChatGPT web settings from 5.2 to 5.5 only. The authors say GEO strategies should be re-evaluated when the deployed model setting or serving environment changes.
- It did not test any commercial AI visibility tool.
- Its importance figures describe associations inside a fitted model. The paper says plainly that changing one feature does not necessarily raise the chance of being cited, and that the results do not imply publishing more pages will help.
What this means for Thai marketers
The following is RA's reading of the paper, not something the authors tested.
If a report claims to show where a brand "ranks" in ChatGPT at the level of single pages, ask how it gets the data. If the answer is Google rankings or one API call per prompt, this paper suggests most of the exact URLs ChatGPT cites will be invisible to it. Site-level and entity-level tracking, run several times per prompt, held up much better in the study and is the more defensible way to report on generative engine optimization work.
The English-only search behaviour deserves attention in Thailand, with a caveat. The paper tested Chinese prompts, not Thai ones, so it does not tell us what ChatGPT does with a Thai question. But if the pattern carries over, a Thai brand that only publishes in Thai could be absent from the English searches ChatGPT runs on a Thai user's behalf. Checking whether key product and service pages have a credible English version is a cheap test, and it fits naturally into a ChatGPT SEO review.
Three working habits follow from the data. Separate "retrieved" from "cited" when you report, because the paper shows they are different stages. Rerun prompts multiple times and track how often a site appears, not whether one page appeared once. And confirm which model answered each run, because silent fallbacks were real in this study. For brands planning broader AI SEO work, the paper's message is that hostname and entity presence is a steadier target than any single URL.
Frequently asked questions
Can Google rankings tell me whether ChatGPT will cite my page?
Not reliably, according to this study. Google's top five pages of organic results contained only 9.4% of the exact URLs cited by GPT-5.3 Instant and 6.4% for GPT-5.5 Instant, although they did better at the site level, at 26.1% and 22.8% of cited hostnames.
Is the OpenAI API a good stand-in for ChatGPT?
It is better than Google but still partial. The API's full source list covered about 22.5% to 22.9% of ChatGPT's cited URLs and about 38% of cited hostnames, while its own citations were more precise, especially against GPT-5.5 Instant.
Did the study test Thai-language prompts?
No. The 80 questions were in English and Chinese only. The finding that GPT-5.3 and GPT-5.5 Instant searched in English only in about 67% to 70% of Chinese-question runs may be relevant to Thai, but the paper does not test it.
Does this mean AI visibility tracking tools are wrong?
The paper did not test any commercial tool, so it does not say that. It shows that two common proxy methods miss most exact cited URLs and do much better at hostname and entity level, which is a reason to ask any tool how its data is collected.
How many times should I run a prompt when checking visibility?
The paper gives no fixed number, but it ran each question 20 times and found that 40.8% to 44.5% of retrieved URLs appeared in only one run. A single check is therefore a weak measurement, and the model serving each answer should be verified.
The study is a useful correction for anyone treating ChatGPT like a ranking table. Pages churn, queries get rewritten, and outside data sees only about a quarter of the exact URLs that get cited. If you want help setting up site-level and entity-level tracking that reflects how ChatGPT actually retrieves sources, Relevant Audience can review your current reporting with you.
