AI Search Indexing: How LLMs Crawl, Learn, and Recall Your Site

First published: 29 June 2026 · Last updated: 29 June 2026

CLASSICAL (Google, Bing)

Page-level indexing

1. Crawler fetches page
2. Parser extracts text, links, schema
3. Indexer stores keywords + signals at page level
4. Ranker scores pages for query relevance + authority
5. Top 10 pages returned as SERP

vs

AI ENGINES (ChatGPT, Perplexity, Claude)

Chunk-level embedding + retrieval

1. Crawler fetches page (own + Bing/Google API)
2. Parser chunks content into 200-500 token segments
3. Embedder converts chunks to vector representations
4. Vector index stores chunks with brand + URL metadata
5. Query embeds → similarity search → top chunks → LLM synthesises answer + citations

The most under-explained question in 2026 SEO is the simplest one: how do AI search engines actually find, read, and remember the content they cite? The conventional wisdom is some variant of "they use Google's index" or "they crawl like Googlebot does". Both are partially true and mostly misleading. The actual answer is structurally different from classical search and the differences matter for content strategy. This article is the technical deep-dive. We work through the AI engine indexing architecture (crawl → chunk → embed → retrieve), the differences between the major engines (ChatGPT, Perplexity, Claude, Gemini, Copilot), the implications for content structure, and the diagnostic methodology for measuring whether your site is being effectively indexed by each engine. The audience is SEO practitioners who want to understand the underlying mechanism, not just the surface-level optimisation rules. For the broader practical context, our existing AI crawlers guide covers the bot landscape and access control, our multi-engine ranking guide covers the per-engine optimisation tactics, and our GEO playbook covers the strategic application. This post drills into the indexing mechanism itself.

How Classical Search Engines Index (Brief Recap)

A short refresher on Google and Bing because the AI engine architecture is best understood as a contrast. Classical search engines (Google, Bing) operate on a four-stage pipeline:

1. Crawl. A bot (Googlebot, Bingbot) fetches the page over HTTP. Discovery happens via sitemaps, internal links, external links, and known URLs. The crawler respects `robots.txt` and crawl-delay directives.

1. Parse. The crawler extracts the rendered HTML (with JavaScript execution for Googlebot, partial for Bingbot), identifies text, headings, links, images, structured data.

1. Index. The parsed content is stored in a massive inverted index keyed by terms (keywords). Each indexed term maps to a posting list of pages containing it, with metadata (term frequency, position, structural role).

1. Rank. At query time, the ranker scores indexed pages against the query using hundreds of signals (keyword match, link authority, freshness, user signals, page experience, E-E-A-T proxies). The top N pages are returned as the SERP.

The critical structural feature: the unit of indexing is the page. The unit of ranking is the page. The unit of return is the page. Google may show featured snippets or AI Overviews that include excerpts, but the underlying ranking decision is page-level.

How AI Search Engines Index

AI search engines operate on a fundamentally different pipeline. The five-stage version:

1. Crawl. A bot fetches the page. The bot stack varies by engine: GPTBot and OAI-SearchBot for OpenAI, ClaudeBot for Anthropic, PerplexityBot for Perplexity, CCBot for Common Crawl (used by many engines as a base layer), Bytespider for ByteDance, Google-Extended for Google's AI training (separate from Googlebot's classical crawl).

1. Parse and chunk. The fetched HTML is parsed and the content is split into semantic chunks. A chunk is typically 200-500 tokens (roughly 150-400 words), bounded by structural breaks (headings, paragraph boundaries, list items). The chunk size is engine-specific but the concept is universal.

1. Embed. Each chunk is converted to a vector representation using an embedding model. The embedding captures the semantic meaning of the chunk in a high-dimensional vector space (typically 768, 1024, or 1536 dimensions). Chunks about similar topics produce similar vectors.

1. Store. The chunks and their embeddings are stored in a vector database alongside metadata: source URL, brand name, publication date, schema-extracted entities, hierarchical context (which H2 section is the chunk under, which page is it on, which site).

1. Retrieve and synthesise. At query time, the user prompt is embedded into the same vector space. A similarity search identifies the top N chunks closest to the prompt. The LLM (GPT-4, Claude, etc.) is given the prompt and the retrieved chunks as context, then generates an answer with the source URLs surfaced as citations.

The architectural difference from classical search is profound. Classical search returns links to pages. AI search synthesises an answer from chunks across multiple pages and surfaces those pages as citations. The unit of indexing in AI search is the chunk, not the page. The retrieval mechanism is semantic similarity, not keyword matching plus link authority.

The Hybrid Reality: AI Engines Use Classical Indexes Too

The above is the pure architecture. The practical reality is hybrid. Most major AI engines combine their own embedding-based index with API access to Google or Bing's classical index for live retrieval. The hybrid pattern by engine:

• ChatGPT (with web search): uses Bing's API for live web search results, plus its own training-time corpus and OAI-SearchBot's lightweight index for supplementary context.
• Perplexity: maintains its own index (PerplexityBot) and supplements with both Google and Bing APIs depending on the query.
• Google AI Overviews and AI Mode: uses Google's classical index directly. The "AI" part is the synthesis layer on top of classical retrieval.
• Microsoft Copilot: uses Bing's classical index directly. Same pattern as Google.
• Claude (Anthropic, with web search): uses Brave Search API for live retrieval plus its own training-time corpus.

The implication: classical SEO still matters because the AI engines are still pulling from classical indexes for live retrieval. A page that ranks well on Google or Bing is more likely to be retrieved into an AI engine's response. The chunk-level optimisation determines whether the retrieved page produces a citation; the page-level optimisation determines whether the page is retrieved in the first place. This is why "SEO is dead" claims are wrong in 2026. The classical SEO surface still feeds the AI engine retrieval pipeline. AEO and GEO are additive layers, not replacements.

Chunking: Why Page Structure Matters Differently

The chunking step is the most consequential difference for content strategy. AI engines do not cite pages; they cite chunks. The chunk's quality determines whether it gets retrieved and cited. What makes a good chunk:

1. Self-contained. A good chunk makes sense in isolation, without requiring context from preceding or following chunks. The fix: write each H2 section as if it could be lifted out and quoted standalone.

1. Topically focused. A chunk that covers one specific question or claim is more retrievable than a chunk that mixes multiple topics. The fix: structure long content so each section addresses one specific question.

1. Densely factual. Chunks rich in extractable facts (numbers, definitions, named entities, dates, comparisons) are preferred for citation because the LLM can lift them verbatim. The fix: structure content with discrete fact units (lists, tables, callouts, definition statements).

1. Schema-tagged when possible. Chunks that are part of a FAQPage, HowTo, or other typed schema get richer metadata in the vector index, improving retrievability. The fix: deploy the schema stack from the AEO content side.

The single most under-appreciated implication: a long-form article that is well-written for human readers but structurally ambiguous (long paragraphs, anaphoric references, mixed topics per section) chunks badly. The same content rewritten with clear sectional boundaries and self-contained answer-units chunks well. Same author, same facts, dramatically different AI citation performance.

High-quality chunk

• Self-contained (200-500 tokens)
• One topic per chunk
• Topic sentence as opener
• Dense factual content
• Within typed schema (FAQPage, HowTo)
• Clear hierarchical context (H2 + parent page)

Retrieved: high. Cited: high. Citation rank: position 1-2.

Low-quality chunk

• Long paragraph blocks (1000+ tokens)
• Mixed topics per section
• Anaphoric references ("as discussed above")
• Prose-heavy, fact-poor
• No schema context
• Weak hierarchical structure (no H2s)

Retrieved: low. Cited: rare. When cited, paraphrased not quoted.

Embeddings: The Semantic Layer

Embeddings are the technical mechanism that makes semantic retrieval possible. A short tour: An embedding is a numerical representation of a chunk's meaning in a high-dimensional vector space. The model that produces the embedding (an embedding model, such as OpenAI's `text-embedding-3-large` or Cohere's `embed-multilingual-v3`) maps each chunk to a fixed-length vector. The vector captures the semantic meaning: chunks about similar topics produce similar vectors, even if the specific words differ. The practical implication: a chunk about "improving page load speed" and a chunk about "Core Web Vitals optimisation" produce similar embeddings even though they share few keywords. A user prompt about "how to make my website faster" embeds close to both. The retrieval step finds the closest chunks regardless of exact-match keyword presence. This is why pure keyword-density SEO does not transfer cleanly to AEO. The retrieval mechanism is not matching keywords; it is matching semantic meaning. The optimisation goal is not to repeat the keyword 12 times; it is to ensure the chunk's semantic content closely matches the kinds of prompts users will run. The corollary: brand-name presence in chunks matters more than keyword density. AI engines match prompts to chunks by semantic similarity but cite the brand attached to the chunk's source. A chunk that comprehensively answers a query but does not mention the brand name will surface the source URL as the citation; a chunk that mentions the brand name in context will additionally surface the brand prominently in the cited answer text.

Per-Engine Indexing Differences

The five major AI engines differ in their indexing details. The summary:

Engine

Primary crawler

Live source

Recency window

Citation density

ChatGPT (search mode)

OAI-SearchBot, GPTBot

Bing API

Days to weeks

3-5 sources per response

Perplexity

PerplexityBot, Perplexity-User

Own + Google + Bing APIs

Hours to days

5-10 sources per response (highest)

Google AI Overviews

Googlebot + Google-Extended

Google classical index

Real-time

3-7 sources per response

Microsoft Copilot

Bingbot

Bing classical index

Real-time

3-5 sources per response

Claude (web search)

ClaudeBot

Brave Search API

Days to weeks

2-4 sources per response (lowest)

The strategic implications:

• For Perplexity: the highest citation density per response means the most opportunities to be cited. Perplexity is the engine to prioritise for AEO measurement because it produces the most signal per query.
• For ChatGPT: dependency on Bing API means Bing rankings matter more than Google rankings for ChatGPT visibility. Bing-specific optimisation (Bing Webmaster Tools verification, Bing-friendly crawl directives) has measurable AEO leverage for ChatGPT.
• For Google AI Overviews: real-time use of the Google classical index means classical SEO is the primary lever. AI Overview citation correlates strongly with traditional ranking on the same query.
• For Copilot: same Bing-dependency as ChatGPT but with stronger Microsoft ecosystem integration (Office, Edge browser). The user base skews enterprise.
• For Claude: lowest citation density per response. Less leverage per query but Claude's user base is heavily technical/developer, making it valuable for B2B SaaS and developer tools verticals.

Recency: How AI Engines Handle Fresh Content

A common misconception is that AI engines index slowly because the underlying LLMs have training cut-offs. The reality in 2026 is that the live-retrieval layer (Bing API, Google index, own crawls) handles freshness independently of the LLM's training cut-off. By engine:

• Real-time engines (Google AI Overviews, Microsoft Copilot): content indexed by Googlebot or Bingbot is available within minutes-to-hours of publication.
• Hours-to-days engines (Perplexity): their own crawler operates continuously and re-indexes priority sources frequently.
• Days-to-weeks engines (ChatGPT, Claude): the proprietary crawls are slower; new content takes longer to enter the AI engine's recall set, even if it is in Bing's index.

The implication for content strategy: time-sensitive content (news, product launches, updates) should be optimised for Google and Bing first because the AI engines that depend on those classical indexes will pick it up quickly. Evergreen content has more time to be ingested by all engines. For SEO practitioners, the freshness signal matters less than for classical search but is not zero. AI engines weight publication date and last-updated date as part of relevance scoring; perpetually stale content gets downranked.

Diagnosing AI Indexing for a Site

The diagnostic methodology for confirming a site is being indexed effectively by AI engines:

1. Crawl confirmation. Check server logs for hits from GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot. Frequency and distribution across pages indicate which content is being prioritised.

1. Inclusion check. For a sample of pages, run direct prompts that should retrieve the page if indexed. Example: "Give me a summary of [page topic] from [site name]". If the AI engine produces a coherent response that matches the page content and cites the page, the page is retrievable.

1. Comparison check. Run the same prompts in incognito for the brand and the top 3 competitors. Note the citation distribution; absence of brand citations against present competitor citations indicates indexing or extractability gaps.

1. Chunk-level audit. Sample 5-10 pages, identify the H2 sections, run prompts that target the specific H2 content. If the H2 is well-indexed, the prompt should retrieve the page; if not, the chunk is not surfacing.

1. Schema verification. Use the Rich Results Test to confirm schema is rendering. AI engines use schema for chunk metadata; chunks within typed schema (FAQPage, HowTo) have richer retrievability.

1. Recency confirmation. Publish a new test article. Track when it first appears as a citation in each engine. The lag identifies the engine-specific recency profile for the site.

The output of this diagnostic is a per-engine indexing health report: which engines are crawling actively, which content is being recalled, where the chunks are weak, and where the schema is missing.

A Worked Example: SG B2B Site Indexing Diagnosis

Concrete example. Client: SG B2B SaaS, 240 indexed pages, AI indexing diagnosis run in March 2026. Crawl confirmation (server logs, 30-day window):

• Googlebot: 4,200 hits across 240 pages.
• Bingbot: 1,800 hits.
• GPTBot: 380 hits.
• ClaudeBot: 220 hits.
• PerplexityBot: 540 hits.
• OAI-SearchBot: 150 hits.

Observation: AI crawlers visit 5-10x less frequently than Googlebot. Concentration is on the top-50 pages by inbound links. Long-tail pages get sparse AI crawler coverage. Inclusion check (15 sampled pages, prompted across 4 engines):

• ChatGPT: 9 of 15 pages produced coherent retrieval; 6 failed.
• Perplexity: 12 of 15 retrieved.
• Google AI Overviews: 14 of 15 retrieved (mirrors classical Google indexing).
• Copilot: 11 of 15 retrieved.

Observation: ChatGPT was weakest, mostly on the long-tail pages with sparse internal links. The fix: improve internal linking (Wikipedia pattern from earlier in this article series). Chunk-level audit (5 pages × 4 H2 sections each):

• 14 of 20 H2 sections were retrievable via targeted prompts.
• 6 failed retrieval, all of them in long paragraph-heavy sections without internal structure.

Observation: chunks need cleaner sectional boundaries. Rewriting the failing sections into self-contained, fact-dense answer-units restored retrieval within 30 days. Day 60 re-test:

• ChatGPT inclusion: 13 of 15 (up from 9).
• Chunk retrieval: 19 of 20 (up from 14).
• AI citation rate on tracked-prompt set: 27% (up from 14%).

The diagnosis-then-fix loop is the right approach. The diagnostic identifies the specific failure modes; targeted fixes produce measurable lift within one indexing cycle.

Frequently Asked Questions

Do AI engines actually crawl my site or just use Google's index?

Most major AI engines do both. ChatGPT, Claude, and Perplexity each operate their own crawlers (OAI-SearchBot/GPTBot, ClaudeBot, PerplexityBot) which ingest content into proprietary embedding indexes used for semantic recall. They additionally call Bing or Google search APIs at query time for live retrieval, particularly for time-sensitive queries. Google AI Overviews and Microsoft Copilot use Google's and Bing's classical indexes directly, so the "AI" layer is the synthesis on top of classical retrieval rather than separate indexing. The implication: every major engine ingests your content somehow, but the path differs.

What is a chunk and why does the size matter?

A chunk is a contiguous segment of content that AI engines treat as a unit for indexing and retrieval. Typical chunk sizes are 200-500 tokens (roughly 150-400 words), bounded by structural breaks like H2 boundaries, paragraph breaks, or list items. The chunk size matters because it is the unit of retrieval: a query is matched against chunks, not whole pages. Content that does not chunk cleanly (because sections are too long, mix multiple topics, or lack structural boundaries) is harder to retrieve and harder to cite. The structural fix is to write content with clear sectional boundaries and self-contained answer-units.

Does keyword density still matter for AI search?

Less than it does for classical search. AI engines match prompts to chunks via semantic embedding, which captures meaning rather than literal word match. A chunk that comprehensively addresses a topic without repeating the exact keyword 12 times can still be retrieved and cited. What matters more: semantic completeness (does the chunk fully address the topic), fact density (does the chunk contain quotable specifics), and brand mention (is the brand named in the chunk's content). Pure keyword-density tactics are rarely the highest leverage; they are also not actively penalised, just superseded.

How do AI engines decide which sources to cite?

The retrieval step ranks chunks by semantic similarity to the prompt. The synthesis step then generates an answer from the retrieved chunks. The citation step links back to the source URLs of the chunks the LLM used most heavily. The factors influencing which chunks get retrieved most heavily: semantic match to the prompt, source authority (entity signals, third-party mentions), recency, schema-typed metadata, and engine-specific preferences (e.g. Perplexity weights freshness more than Claude does). The factors influencing which retrieved chunks get cited prominently: how closely the chunk's content matches the user's specific prompt phrasing and how complete the answer is within a single chunk.

Why is ChatGPT slower to index new content than Perplexity?

Two reasons. First, OpenAI's proprietary crawl rate (OAI-SearchBot, GPTBot) is lower than Perplexity's PerplexityBot, so new content enters the proprietary embedding index more slowly. Second, when ChatGPT relies on Bing API for live retrieval, the lag from publication to Bing index inclusion is the bottleneck (typically 1-7 days for a non-priority site). Perplexity's higher own-crawl rate plus its blended use of Google API for live retrieval reduces both lag sources. The practical implication: time-sensitive content has higher AEO leverage on Perplexity, AI Overviews, and Copilot than on ChatGPT or Claude.

How do I know if my content is well-chunked?

The simplest test: take any H2 section of your page and read it cold, without the surrounding context. Does it answer a question? Does it stand alone? Does a reader unfamiliar with the rest of the page understand it? If yes, it chunks well. If the section requires references to earlier or later sections to make sense ("as we discussed above", "we will cover this in detail later"), it chunks poorly. The structural fix is to rewrite each section to be self-contained: the topic sentence states the answer, supporting sentences provide evidence, no anaphoric references. This is the single highest-leverage rewrite for chunk quality.

Related reading

• AI crawlers guide: per-bot directives, robots.txt patterns, access control
• Multi-engine ranking: per-engine optimisation tactics for ChatGPT, Perplexity, Claude, AI Overviews
• GEO optimisation playbook: applying the indexing knowledge to a content programme