How AI answer engines decide which sources to cite

The gist

There is no single algorithm. ChatGPT, Perplexity, Google’s AI Overviews, Gemini, and Claude each retrieve and cite in their own way. But underneath, they share a pattern: they pull candidate pages from a search index, then a model writes an answer and cites the handful it leaned on. The pages that get cited are, with striking consistency, the ones that already rank, are easy for a crawler to read, carry real authority, and answer the question in plain, extractable text. That is the target — not a trick aimed at any one engine.

Most of them don’t have their own index

The first thing to understand is that AI answer engines mostly ride on a traditional search index rather than crawling the whole web themselves. ChatGPT Search retrieves largely from Bing, increasingly supplemented by its own search crawler and licensed feeds. Microsoft Copilot uses Bing. Google’s AI Overviews and AI Mode, and Gemini’s grounding, use Google Search. Claude’s web search appears to be backed by Brave — Anthropic added Brave Search to its subprocessor list, and independent testing turned up identical citations and a BraveSearchParams field in Claude’s search tool.

The practical consequence is blunt: being indexed and ranking in the underlying engine is the price of admission. If Bing or Google can’t find you, the AI built on top of it can’t cite you. This is why “AI search made SEO obsolete” is exactly backwards — classic indexing is the foundation the whole thing stands on.

The citation is chosen before the prose is written

Perplexity is the clearest example of the shape. It retrieves candidate documents, reranks them, and only then has a model synthesize an answer with inline citations constrained to what it retrieved. Switching the model behind a Pro answer changes how the answer reads, not which documents were pulled.

Google’s AI Mode does something it calls query fan-out: it breaks your question into subtopics and issues a multitude of searches simultaneously, then synthesizes across the results. Gemini decides whether to search at all, and when it does, the grounding response carries a groundingMetadata object listing the webSearchQueries it ran and the groundingChunks it used. Claude treats web search as a tool it chooses to call, then cites inline.

The lesson under all of them is the same: the source is chosen at retrieval time, from an index, before any sentence is generated. You are not optimizing a paragraph after the fact. You are trying to be in the candidate set.

What actually gets a page pulled and cited

Across engines, the levers converge on a short list.

• Be in the index and rank for the query. Retrieval starts from the same search index you have always tried to rank in. No ranking, no candidacy.
• Be readable by a crawler. AI retrieval crawlers mostly do not run JavaScript — Vercel and MERJ analyzed 569 million GPTBot requests across their network and found none of the major AI crawlers render JavaScript at all; they fetch raw HTML and skip client-side rendering. A single-page app that paints its content after hydration is an empty page to them. Server-render so the content is in the first response.
• Be extractable and answer-first. Lead with the answer and state claims plainly. The original GEO research (KDD ‘24) found that adding cited sources, statistics, and quotations raised a page’s visibility in generative answers by up to about 40%, and that the lift helps pages outside the top spot the most. The behavior shows up in the wild too: Ahrefs’ study of 1.4 million ChatGPT prompts found that whether a page gets cited tracks how well its title and URL line up with the specific sub-questions ChatGPT fans out — descriptive, answer-shaped pages won the citation more often.
• Carry authority as an entity. A consistent name and description across the web, and third-party mentions, make a source easier for a model to trust.
• Be structured. JSON-LD per Google’s structured-data docs makes your facts machine-readable instead of inferred from layout.
• Be fresh. On time-sensitive topics, recently updated content is favored — but only if the body actually changed, not the date stamp.

A quirk worth knowing about

There is a mechanical bias in how neural retrievers score relevance: they tend to over-favor low-perplexity text — fluent, predictable, well-edited writing. A causal study published at ICLR 2025 showed that retrievers learn to treat low perplexity as a proxy for relevance, which can bias them toward smooth, machine-friendly prose.

The honest takeaway is not to try to game it. It is that clear, fluent, well-structured writing is rewarded twice — once by the human reading it, once by the retriever scoring it. The thing that makes a page good for a person is, conveniently, the thing that makes it easy to retrieve.

What it doesn’t reward

The shortcuts do not survive contact with these systems. Keyword stuffing tested worse than the baseline in the GEO study. SE Ranking analyzed about 300,000 domains and found no correlation between having an llms.txt file and how often a domain gets cited — Google’s John Mueller says plainly that no AI service currently fetches it. And Google has folded manipulating generative AI answers into its spam policies, which now define spam to include “attempting to manipulate generative AI responses in Google Search.”

Citations track substance, not manipulation. That is the whole reason the durable play is the same unglamorous, white-hat checklist: be crawlable, be authoritative, be extractable, be fresh, and be worth quoting.

Why this is worth doing now

The audience for all of this is growing fast. Gartner forecasts that by 2028, 90% of B2B buying will be AI-agent intermediated, pushing more than $15 trillion of B2B spend through agent exchanges. As more buyers arrive through an AI answer rather than a ranked link, being one of the sources those answers are built from stops being optional. The mechanics differ by engine; the work to earn a citation does not.

References

1. Vercel & MERJ — The rise of the AI crawler. Analysis of 569 million GPTBot requests finding that no major AI crawler renders JavaScript (Vercel, 2024).
2. Yoast — What is ChatGPT Search (and how does it use Bing data)?. Explainer on ChatGPT Search retrieving from the Bing index (Yoast).
3. TechCrunch — Anthropic appears to be using Brave to power web search for Claude. Subprocessor list, identical citations, and a BraveSearchParams field point to Brave (Whittaker, 2025).
4. Google — AI Mode in Search: updates from Google I/O 2025. Official description of the query fan-out technique (Google, 2025).
5. Google — Grounding with Google Search. Gemini API docs on the groundingMetadata, webSearchQueries, and groundingChunks returned with a grounded answer (Google).
6. Aggarwal et al. — GEO: Generative Engine Optimization (KDD ‘24). The original GEO study; adding cited sources, statistics, and quotations lifts visibility in generative answers by up to ~40%.
7. Ahrefs — Why ChatGPT cites one page over another (study of 1.4M prompts). Citation tracks how well a page’s title and URL align with ChatGPT’s fan-out sub-questions (Linehan, Ahrefs).
8. Cheng et al. — Perplexity Trap: PLM-Based Retrievers Overrate Low Perplexity Documents (ICLR 2025). Causal study showing neural retrievers treat low perplexity as a proxy for relevance.
9. SE Ranking — Does llms.txt affect AI visibility?. Analysis of ~300,000 domains finding no correlation between llms.txt and AI citations (SE Ranking).
10. Search Engine Roundtable — Google: no AI system currently uses llms.txt. John Mueller’s Reddit/Bluesky comments that AI services don’t fetch the file (Schwartz, 2025).
11. Google — Spam policies for Google Web Search. Spam now includes “attempting to manipulate generative AI responses in Google Search” (Google).
12. Gartner — Gartner Unveils Top Predictions for IT Organizations and Users in 2026 and Beyond. Prediction that by 2028, 90% of B2B buying will be AI-agent intermediated, pushing over $15 trillion of B2B spend through agent exchanges (Gartner, 2025).
13. Google — Intro to structured data markup (JSON-LD). Official documentation on making facts machine-readable (Google).