AI crawlers (like GPTBot, PerplexityBot) fetch and parse web pages similarly to search crawlers, but the retrieval-augmented engines (Perplexity, and partly Gemini) also pull live at query time, not just from a pre-built index.
[01]/ the data
What we found
This is why Perplexity citation counts shift week to week in our tracking, while ChatGPT/Claude citations (trained more on static data) shift much more slowly.
[02]/ how to do it
Three steps
Check your robots.txt allows the relevant bots.
Publish an llms.txt file describing your site for AI consumption.
Keep key pages freshness-dated, since retrieval-based engines reward recency.