Searcharxiv⌕ Search
Home/Archive/arXiv

arXiv · 2610.15382

Understanding ChatGPT Citations: Evidence from Production Web Search Across Model Generations

Yunxuan Jiang·Shiqi Zhang·Anqi Xing·Xiangyu Chang·Xiaokui Xiao
View original source ↗Read source document ↗Export metadata ↓

Abstract

Generative engines such as ChatGPT Web retrieve webpages to answer user queries, but only a subset are shown as citations. Understanding this retrieval-to-citation process is important for generative engine optimization (GEO), which aims to improve content visibility in generative-engine responses. However, prior studies mainly analyze final responses, leaving the internal execution process and its variation across model settings less understood. To this end, we conduct an observational study of four ChatGPT Web settings from ChatGPT 5.2 to 5.5, covering Instant and Thinking modes at the URL, hostname, and entity levels. We examine how ChatGPT rewrites queries, retrieves candidate webpages, and selects references. The Instant settings remain mostly single-round, while GPT-5.4 Thinking frequently performs multiple retrieval rounds. Across repeated runs, candidates retrieved more often are also more likely to be cited. We then evaluate Google Search and the OpenAI Responses API as external proxies. Responses API candidates provide broader coverage, while its citations more closely match sources cited by ChatGPT Web. Both proxies match ChatGPT Web better at the hostname and entity levels than at the exact-URL level. Even when combined, they recover only about one quarter of cited URLs. We further test whether these external signals remain predictive when combined. Within the external candidate set, a supervised model combining them reaches AUCs above 0.74 for GPT-5.3 and GPT-5.5 at all three levels. These findings suggest that GEO should distinguish retrieval visibility from citation visibility, account for rewritten queries and run-to-run variation, and consider hostname- and entity-level presence in addition to exact URLs. Differences across settings also suggest that GEO strategies should be re-evaluated when the deployed setting or serving environment changes.

Explore related subjects

cs.IR

Keep this discovery

Explore connections, maps & timelines

BibTeXRIS

Yunxuan Jiang, Shiqi Zhang, Anqi Xing, Xiangyu Chang, Xiaokui Xiao. 2026-10-11. Understanding ChatGPT Citations: Evidence from Production Web Search Across Model Generations. https://arxiv.org/abs/2610.15382

Cite the original work for its findings. Save a collection to share your selection of sources.

KEEP EXPLORING

Related papers

arXiv2026

EviRerank: Adaptive Evidence Construction for Long-Document LLM Reranking

Decoder-only LLM rerankers struggle with long documents: inference is costly and relevance signals can be diluted by irrelevant context. Motivated by an attention analysis indicating a consistent degradation trend when non-relevant text is appended, we propose EviRerank, an evidence-based long-document reranking framework for decoder-only LLMs. EviRerank (i) scores document blocks with a lightweight selector (BM25, bi-encoder, or cross-encoder), (ii) constructs a compact reranking context under a hard token cap by dynamically budgeting evidence blocks with Adaptive Evidence Budgeting (AEB) and adding a global summary cue via Summary Augmentation (SA), and (iii) reranks with a decoder-only LLM. Across TREC DL'19, DL'22, DL'23, and MLDR-zh, EviRerank consistently outperforms full-document LLM reranking and strong block-selection baselines while substantially reducing the required input length. On TREC DL'19, EviRerank reaches up to 0.744 nDCG@10 while substantially reducing reranking latency. Additional RankZephyr-7B experiments confirm transfer to listwise reranking under matched context budgets.

cs.IR↗
arXiv2026

On the Impact of Graph Neural Networks in Recommender Systems: A Topological Perspective

In recommender systems, user-item interactions can be modeled as a bipartite graph, where user and item nodes are connected by undirected edges. This graph-based view has motivated the rapid adoption of graph neural networks (GNNs), which often outperform collaborative filtering (CF) methods such as latent factor models, deep neural networks, and generative strategies. Yet, despite their empirical success, the reasons why GNNs offer systematic advantages over other CF approaches remain only partially understood. This monograph advances a topology-centered perspective on GNN-based recommendation. We argue that a comprehensive understanding of these models' performance should consider the structural properties of user-item graphs and their interaction with GNN architectural design. To support this view, we introduce a formal taxonomy that distills common modeling patterns across eleven representative GNN-based recommendation approaches and consolidates them into a unified conceptual pipeline. We further formalize thirteen classical and topological characteristics of recommendation datasets and reinterpret them through the lens of graph machine learning. Using these definitions, we analyze the considered GNN-based recommender architectures to assess how and to what extent they encode such properties. Building on this analysis, we derive an explanatory framework that links measurable dataset characteristics to model behavior and performance. Taken together, this monograph re-frames GNN-based recommendation through its topological underpinnings and outlines open theoretical, data-centric, and evaluation challenges for the next generation of topology-aware recommender systems.

cs.IR↗
arXiv2026

Relevance Supervision Can Teach Neural Retrievers to Prefer LLM-Generated Text

Recent studies show that neural retrievers often exhibit source bias, ranking LLM-generated passages above semantically similar human-written ones, which threatens the reliability of modern information access systems. While this bias is often viewed as a model-level property, we show that relevance supervision itself can induce it. Empirically, source bias is clearest in relevance-supervised retrievers, and the same encoders become more LLM-preferring after relevance fine-tuning. We trace this effect to surface-level artifacts in supervised retrieval data: annotated positives and negatives differ systematically in properties such as fluency and lexical specificity, and these differences parallel those between LLM-generated and human-written passages. As a result, retrievers can learn artifact-based shortcuts for relevance. We further show theoretically that, under contrastive learning, artifacts that reliably separate positives from negatives can be encoded into learned relevance scores. Motivated by this mechanism, we propose complementary strategies that reduce source bias during training and at inference: balancing artifact distributions in negative sampling and removing the artifact-aligned embedding component, respectively. Overall, our findings support a data-centric view of source bias: retrievers may prefer LLM-generated text not merely because of inherent model properties, but because relevance supervision teaches them to rely on surface-level artifacts.

cs.IR↗