A growing share of searches now end in an AI-generated answer: Google AI Overviews, ChatGPT search, Perplexity, Copilot. These answers usually cite a handful of sources. Being one of those sources is the new version of ranking on page one. The fundamentals overlap heavily with good SEO, but there are a few differences worth knowing.
How AI search finds sources
Most AI search products do not answer purely from memory. They run a live search, fetch a set of relevant pages, and write an answer grounded in those pages, citing them as they go.
- Google AI Overviews draw on Google's own index, so normal Google SEO is the foundation.
- ChatGPT search uses its own crawler, OAI-SearchBot, together with third-party search providers, which is one reason Bing Webmaster Tools is worth setting up.
- Perplexity crawls with PerplexityBot and also fetches pages live when answering.
- If your page is not indexed, crawlable and quick to fetch, it cannot be cited.
1. Decide which AI crawlers to allow
AI companies use separate user agents for training models and for live search. You can block training while still allowing your pages to be found and cited.
- OAI-SearchBot: OpenAI's crawler for ChatGPT search results.
- GPTBot: OpenAI's crawler for model training.
- PerplexityBot: Perplexity's search crawler.
- ClaudeBot: Anthropic's crawler.
- Google-Extended: controls whether your content is used for Gemini model training. It does not affect Google Search or AI Overviews.
# Allow AI search crawlers so pages can be cited
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
# Optional: opt out of model training
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xml2. Make content visible without JavaScript
Googlebot renders JavaScript, but many AI crawlers fetch raw HTML and move on. If your main content only appears after scripts run, they may see an empty page. Server-side rendering or static generation puts your words in the initial HTML for every crawler.
Quick test
3. Write content that is easy to cite
- Answer first: put a direct one or two sentence answer right below each question-style heading.
- Be specific: concrete numbers, thresholds, steps and dates are easier to quote than general advice.
- Use clear headings that match how people phrase questions.
- Show who wrote it and when it was updated. Freshness and expertise signals matter for both search and AI answers.
- Cite your own sources. Content that links to primary research tends to be treated as more trustworthy.
- Publish original information, such as your own data, benchmarks or case studies, that cannot be found elsewhere.
4. Add an llms.txt file (cheap, optional)
llms.txt is a proposed standard from 2024: a plain-text Markdown file at the root of your site summarizing what you do and linking to your most important pages. It is quick to add and some AI tools read it. Be realistic though: major search engines have not confirmed they use it for ranking or citations, so treat it as a bonus, not a strategy.
# Example Co
> Example Co builds fast, accessible websites for local businesses.
## Key pages
- [Services](https://www.example.com/services): What we build and for whom
- [Pricing](https://www.example.com/pricing): Plans and what is included
- [Guides](https://www.example.com/blog): Practical articles on web performance5. Build a clear brand entity
AI systems lean on what the wider web says about you. Consistent Organization structured data, matching social profiles, and genuine mentions on relevant sites all help models understand who you are and when you are relevant to a question.
6. Keep pages fast
When an AI assistant fetches pages live to answer a question, slow or failing pages are easy to skip. Fast server responses and lightweight HTML make your page a reliable source, every time.
How to measure AI visibility
- Check referral traffic in your analytics from chatgpt.com, perplexity.ai and similar domains.
- Search your key questions in each AI tool regularly and note whether you are cited.
- Look for AI crawler user agents in your server logs to confirm they can reach your pages.
- Track Search Console impressions, which include searches where AI Overviews appear.
The sites AI search cites most are the ones that were already easy to crawl, fast to load and genuinely useful. LightLoader takes care of the speed part. The rest is good content, made easy to find.