First published: 29 July 2026 · Last updated: 29 July 2026
Crawl & Index
Dedicated crawlers (GPTBot, ClaudeBot, PerplexityBot) supplemented by licensed datasets (Common Crawl, Bing Index, Google Index, partner feeds). Content stored as embeddings + raw text for retrieval.
Retrieve & Score
At query time, retrieve candidate passages via semantic search (embedding similarity) plus keyword matching. Score each passage for relevance, authority, specificity, freshness. Passages, not pages, are the unit.
Synthesise & Cite
Rank passages by composite score. Highest-scoring chunks get quoted or paraphrased in the answer. Sources cited via inline links, footnotes, or sidebar references depending on engine surface.
The Foundational Research: GEO by Aggarwal et al
The academic foundation for GEO as a discipline is "GEO: Generative Engine Optimization" by Pranjal Aggarwal et al., published at ACM SIGKDD 2024. The paper is a joint effort from IIT Delhi, Princeton University, Georgia Tech, and the Allen Institute for AI. It introduced GEO-bench, a benchmark of 10,000 diverse queries, and systematically tested nine optimisation strategies against generative engine response visibility. The headline finding: optimisation strategies produced visibility lifts of up to 40% over baseline content. The most effective strategies: adding authoritative citations, statistics, and fluency improvements lifted scores from a baseline of 19.3 visibility metric to over 40, a >100% improvement. Tables earned approximately 2.5x more AI citations than the same information presented as prose. Direct answers in the first 40-60 words of a section significantly increased citation probability. These findings established the empirical foundation for the GEO tactical playbook that has matured through 2024-2026. Every recommendation in this programme traces back to either the Aggarwal paper specifically or the broader pattern of empirical findings the paper crystallised. The principles are not opinion; they are measurable empirical patterns observed across thousands of queries against multiple AI engines. The implication for practitioners: GEO is not a speculative discipline. The optimisation principles have empirical backing comparable to what classical SEO had in its early academic literature (Brin and Page, 1998; subsequent work on PageRank variants). The discipline is younger but the methodology of empirical validation is in place.Stage 1: Crawl and Index
The crawl and index stage is where AI engines acquire the content corpus they later retrieve from. The mechanisms vary by engine but follow a shared pattern. Dedicated crawlers are the most visible component. GPTBot crawls for OpenAI training, ChatGPT-User for OpenAI live retrieval, OAI-SearchBot for ChatGPT Search index, ClaudeBot for Anthropic training, Claude-User for Anthropic live retrieval, PerplexityBot for Perplexity index, Perplexity-User for live retrieval. Each crawler has its own user agent, IP range, and crawl cadence. The dedicated crawlers are the AI-engine-specific layer of the corpus. Licensed and partner datasets supplement the dedicated crawls. OpenAI's models train on Common Crawl data, licensed news content, and partner data feeds. Anthropic's Claude models similarly draw from licensed corpora. Perplexity uses live web retrieval supplemented by indexed snapshots. Google AI Overviews draw directly from Google's primary search index. The licensed and partner datasets often dominate volume; the dedicated crawls fill the gaps and refresh the freshest content. Bing Index is the critical infrastructure layer for ChatGPT and Microsoft Copilot. ChatGPT live retrieval queries Bing for current information; ChatGPT shopping carousels pull heavily from Bing's product data. The implication: Bing SEO is now an AEO discipline because the Bing index is the live retrieval layer for OpenAI's ecosystem. Sites with poor Bing presence systematically underperform in ChatGPT live retrieval. Google Index plays the analogous role for Google AI Overviews. Pages well-indexed in Google's primary search are eligible candidates for AIO source citation. Pages poorly indexed are not. Classical Google SEO directly affects AIO eligibility. Vector embeddings storage. AI engines do not store crawled content as raw text only. They generate vector embeddings for each chunk of content (typically passage-level chunks of 100-500 tokens) and store the embeddings in vector databases for semantic similarity retrieval. The embedding generation step is where content quality, structure, and semantic clarity translate into retrievable vectors. The implication for SEO: classical crawl optimisation (sitemaps, robots.txt, internal linking, crawl budget management) remains foundational for AI engine corpus inclusion. The new layer is content structure that produces clean chunk boundaries and semantically dense embeddings; this is the chunkability principle that runs through the AEO content framework.Stage 2: Retrieve and Score
The retrieve-and-score stage is where AI engines select candidate passages from the indexed corpus for inclusion in the answer synthesis. This is where the chunk-level optimisation work pays off or fails. Query parsing and decomposition. When a user asks a question, the engine first parses the query and often decomposes it into sub-queries. "What is the best CRM for Singapore SMBs and how much does it cost" might decompose into "best CRM for Singapore SMBs" and "CRM pricing for SMBs". Each sub-query is processed separately for retrieval. Embedding-based semantic search. Each sub-query is embedded into a vector representation. The engine performs nearest-neighbour search against the indexed embeddings to find the top-N candidate passages by semantic similarity. Typical N: 50-200 candidate passages per sub-query. Keyword and entity matching. In parallel with semantic search, the engine performs lexical matching against keywords and named entities in the query. Passages that match both semantically and lexically score higher than passages that match only one. Per-passage scoring. Each candidate passage receives a composite score based on:- Relevance: how well the passage answers the specific sub-query.
- Authority: signals from the source domain (PageRank-equivalent, brand recognition, citations elsewhere).
- Specificity: how directly the passage addresses the question vs how generally.
- Freshness: recency of publication or update, weighted by query type (recent news vs evergreen reference).
- Format suitability: passage structure (direct answer, list, table) matched to query type.
- Schema reinforcement: structured data signals that confirm the passage is the relevant answer type.
Stage 3: Synthesise and Cite
The synthesis stage is where the engine assembles the final answer from the highest-scoring passages and attaches source citations. Passage ranking and selection. From the candidate pool (typically 50-200 passages per sub-query), the engine selects the top 5-15 passages for inclusion in the synthesis. The selection is based on the composite score plus diversity considerations across sources. Answer generation. The selected passages are provided as context to the underlying language model (GPT-4 family for ChatGPT, Claude family for Claude, Gemini family for Google AIO, custom models for Perplexity). The model generates a coherent answer that synthesises across the passages, often quoting key facts directly and paraphrasing synthesis. Citation attachment. Each fact or quote in the generated answer is attached to its source passage. The citation surface varies by engine: ChatGPT shows inline numbered citations with source links, Perplexity shows source cards in a sidebar, Google AIO shows source links inline and below the AI summary, Claude shows source links inline when web search is enabled. Quality filtering. Engines apply post-generation quality filters: factual coherence checks, source agreement checks, hallucination detection. Passages that conflict with the consensus across multiple sources may be down-weighted or excluded. The implication: a single contrarian passage rarely makes it through if multiple authoritative sources contradict. Personalisation and conditioning. Some engines apply user-context conditioning: prior conversation context, user profile, geographic location. Two users asking the same query may receive subtly different answers and source sets. This is the source of much of the tracking complexity in AEO measurement. The synthesis stage is opaque to external observation; we cannot directly observe which passages the engine selected and how they were ranked. We can only observe the final answer and the citations. The work of GEO is to optimise upstream (at the crawl and chunk levels) so the synthesis stage selects our content; we cannot intervene at the synthesis stage itself.Engine-Specific Variations That Matter
The engines share the three-stage pipeline but differ in implementation specifics that matter for tactical optimisation:
ChatGPT. Crawl via GPTBot for training, Common Crawl for backfill, Bing Index for live retrieval. Heavy reliance on Bing for current information makes Bing SEO critical for ChatGPT live citation. Citation surface is inline numbered with source links. ChatGPT Shopping pulls 83% of carousel data from Google Shopping feed.
Claude. Crawl via ClaudeBot for training, Brave Search for live retrieval as of mid-2026. Brave Search optimisation is the analogous critical channel for Claude live citation. Citation surface is inline links shown when web search is enabled by user.
Perplexity. Crawl via PerplexityBot for own index, supplemented by live multi-source retrieval (Bing, Google, others). Most aggressive live retrieval cadence among major engines. Citation surface is sidebar source cards making citations highly visible to users. Perplexity tends to favour authoritative sources with frequent updates.
Google AI Overviews. Crawl via Googlebot for the Google Index that AIO retrieves from. Classical Google SEO is the foundation for AIO eligibility. Citation surface is inline source links plus a "show more" expansion. AIO triggering is highly query-dependent; not all queries get an Overview.
Gemini. Heavy Google Index integration plus licensed data. Often surfaces similar content patterns to AIO. Citation patterns evolving rapidly through 2026.
Microsoft Copilot. Bing Index direct, similar architecture to ChatGPT live retrieval. Citation surface is inline numbered. The Copilot ecosystem is increasingly relevant for enterprise contexts where Microsoft 365 integration matters.
The tactical implication: per-engine optimisation requires knowing which underlying index each engine retrieves from. Bing for ChatGPT and Copilot. Brave for Claude. Google for AIO and Gemini. Multi-source for Perplexity. The work that lifts visibility in the underlying index lifts visibility in the engines that retrieve from it.
The GEO Optimisation Principles That Transfer Across Engines
The principles that hold across all engines because they reflect the shared three-stage pipeline:
Principle 1: Chunkability. Content structured as self-contained, fact-dense, schema-tagged sections produces better embeddings and scores higher in retrieval. Run-on prose, heavy intro-and-conclusion structure, and dense paragraphs without clear chunk boundaries underperform.
Principle 2: Direct answer placement. The first 40-60 words of a section addressing a question should answer the question directly. The Aggarwal paper specifically found this lifted citation probability significantly. The pattern transfers across engines because all engines score the opening of a section higher in relevance assessment.
Principle 3: Tabular and list formatting where appropriate. Tables earn approximately 2.5x more AI citations than the same information in prose. Lists outperform paragraphs for enumerative information. Format-match the query type: comparison queries get tables, process queries get ordered lists, definition queries get direct paragraphs.
Principle 4: Authoritative citations and statistics. Adding authoritative citations and statistics to content lifts visibility scores significantly per the Aggarwal findings. The mechanism: AI engines weight passages with verifiable external citations as more trustworthy and synthesise them more often.
Principle 5: Schema enrichment for parseability. FAQ, HowTo, Product, Article, Organization schema make content explicitly parseable for the AI engines. Pages without schema lose eligibility for many citation types.
Principle 6: Brand and authority signals. AI engines weight passages from authoritative sources higher in the scoring stage. Domain authority, citation count, brand recognition, and editorial reputation all factor into the authority score component.
Principle 7: Freshness for time-sensitive content. Recently updated content scores higher for queries where freshness matters. The dateline and publication date signals are not optional metadata; they affect ranking.
Principle 8: Cross-engine consistency. Content that appears consistently across the engines benefits from cross-validation effects in synthesis. Sites that show up in ChatGPT, Perplexity, AND AIO citations build cumulative authority that compounds.
These eight principles are the foundation. Tactical execution varies by engine and by query type, but the principles transfer. Optimisation work that violates these principles fails on every engine; optimisation work that respects them succeeds across every engine.
What Practitioners Cannot Yet Optimise Directly
The honest limitations of what GEO practitioners can affect:
The synthesis stage itself. We cannot observe which passages the engine selected, how they were ranked, or why specific synthesis decisions were made. Optimisation must happen upstream at the crawl, chunk, and authority levels.
Per-user personalisation effects. Two users asking the same query may receive different answers based on profile, location, and conversation history. We cannot optimise for personalisation effects we cannot observe.
Engine-internal weighting evolution. The relative weights of relevance, authority, specificity, freshness, and format suitability shift over time as engines tune their models. We optimise for stable principles; specific tactical lifts may compress or expand as weights shift.
Hallucination correction and quality filters. The post-generation quality filters that exclude or down-weight conflicting passages are opaque. We cannot directly intervene in this layer.
Live retrieval triggering. Whether a query triggers live retrieval vs uses cached training data is engine and query-dependent. We can structure for both scenarios but cannot force the engine choice.
The gap between what we can optimise and what affects outcomes is real and worth being honest about. The discipline is to focus effort where leverage exists (the eight principles above) and accept that residual variation is part of the territory.
The Strategic Implication for SEO Programmes Through 2027
The synthesis of everything in this article and the previous 44 in the programme:
The technical foundations of AI search are stable enough to optimise against. The Aggarwal research crystallised the empirical principles in 2024; subsequent industry data has consistently validated the same patterns through 2025-2026. The discipline is mature enough for confident tactical execution.
The engines will continue to evolve, and the principles will hold. Specific engines may rise or fall; specific tactics may compress or expand in effectiveness; the underlying chunk-level retrieval architecture is the same architecture that has powered LLM-based question answering since GPT-3 deployment. The principles that fit that architecture will continue to fit subsequent variations.
Per-engine optimisation will become a discipline category alongside technical SEO and content strategy by end of 2027. The teams investing in this category now will own the share-of-voice when the executive citation question becomes standard quarterly reporting in 2027.
The unified content discipline that wins SERPs and citations is converging. By end of 2027, the editorial pattern that captures featured snippets and the pattern that captures AI Overview citations is the same pattern. The work is unified-track; the outcomes are dual-surface.
SEO does not die. SEO becomes the foundational subset of a larger discipline that includes AEO and GEO. Classical SERP optimisation continues to matter because (a) classical search holds majority query share even at end of 2027, (b) AI engines pull live retrieval from classical indexes (Bing for ChatGPT, Brave for Claude, Google for AIO), and (c) structural patterns that win SERPs heavily overlap with patterns that win AI citations.
This is the technical foundation that anchors the strategic outlook in our programme-closing strategic essay. The architecture is comprehensible, the principles are actionable, and the future is substantially predictable from the present-day understanding of how the engines work.
Frequently Asked Questions
How does the Aggarwal GEO paper relate to subsequent industry research?
The Aggarwal et al paper (SIGKDD 2024) established the methodological framework for empirical GEO research. Subsequent work by industry teams (Profound, AthenaHQ, Ahrefs research, Semrush studies, Seer Interactive's CTR studies, ProductRank.ai analyses) has consistently validated the core findings while extending the empirical base. The 2.5x table-vs-prose citation lift, the >100% lift from citations and statistics, the importance of the first 40-60 words: all replicated across larger and more recent datasets. The discipline has empirical depth that classical SEO took a decade to accumulate; GEO accumulated comparable depth in 18-24 months.
Why do AI engines score chunks rather than pages?
The technical reason: vector embeddings work at chunk level because dense embeddings of large documents lose semantic precision. A 5,000-word page condensed into one embedding cannot represent the diverse semantic content; chunked into 20-50 passages of 100-500 tokens each, the embeddings preserve specificity. The retrieval engine matches query embeddings against chunk embeddings, returning the specific chunks that match. This is a fundamental architectural choice across all major LLM-based retrieval systems and is not likely to change because the alternative (whole-document embedding) loses too much precision for question answering applications.
Can I optimise for one engine without affecting others?
Mostly no. The eight principles transfer across engines because they reflect the shared three-stage pipeline. Engine-specific optimisation (Bing for ChatGPT live, Brave for Claude, schema for AIO) shifts the tactical layer, but content that fundamentally violates the principles fails on every engine, and content that respects them tends to succeed across engines. The per-engine work is mostly about coverage (ensuring you appear in each underlying index) rather than fundamentally different optimisation approaches.
How often do AI engines update their models and how does that affect SEO?
Major model updates: roughly every 6-12 months for headline models (GPT-4 to GPT-5, Claude 3 to Claude 4, Gemini iterations). Minor updates and tuning: continuous. Index refresh cadence: continuous for live retrieval, monthly to quarterly for training data refreshes. The implication: short-term tactical lifts may compress or expand with updates, but the underlying principles persist because they reflect the architecture not specific model weights. Practitioners should not panic over individual update windows; they should optimise for the stable principles.
Is there a difference between "AI search" and "generative search"?
The terms are used somewhat interchangeably in 2026. "AI search" is the broader umbrella that includes traditional search engines with AI features (Google AI Overviews, Bing Copilot integration). "Generative search" or "generative engine" specifically refers to engines that generate synthesised answers from retrieved content (ChatGPT, Perplexity, Claude, Gemini). The optimisation discipline (GEO) covers both but with different tactical emphasis. The lines blur as Google integrates more generative features and as ChatGPT integrates more search-like features.
What is the highest-leverage single change I can make to improve AI engine citation today?
Restructure the first 40-60 words of every major section to provide a direct, complete answer to the question that section addresses. The Aggarwal research, multiple industry studies, and our own client portfolio data all converge on this as the single highest-leverage tactical change. The work is moderate (rewrite section openings across the priority pages); the lift is consistent across engines and across query types. If you do nothing else from the eight principles, do this.