# How ChatGPT, Perplexity and other AI search tools pick their sources

Source: https://trynewsusa.com/how-ai-search-picks-sources
Published by NewsUSA. Updated 2026-11-14.

> Short answer: None of them publishes a formula. Their documentation says the same few things: a page has to be open to the tool's search crawler (OAI-SearchBot for ChatGPT, PerplexityBot for Perplexity, Claude-SearchBot for Claude), and indexed by the search behind the answers. Google says "There are no additional requirements to appear in AI Overviews or AI Mode" beyond being indexed and eligible for a snippet. Perplexity and Claude say their answers cite their sources.

## ChatGPT search

- OpenAI runs three crawlers with different jobs. "OAI-SearchBot is used to surface websites in search results in ChatGPT's search features."
- Blocking it keeps you out: "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links." OpenAI says "we recommend allowing OAI-SearchBot in your site's robots.txt file."
- GPTBot is the training crawler: "It is used to crawl content that may be used in training our generative AI foundation models."
- ChatGPT-User fetches a page when a user's request needs it, and "ChatGPT-User is not used to determine whether content may appear in Search."
- The settings are separate: "Each setting is independent of the others." A site can block training and still allow search.

## Perplexity

- "PerplexityBot is designed to surface and link websites in search results on Perplexity." Perplexity also says "we recommend allowing PerplexityBot in your site's robots.txt file."
- Perplexity-User works on demand: "When users ask Perplexity a question, it might visit a web page to help provide an accurate answer and include a link to the page in its response."
- Its help center says "it uses advanced AI to search the internet in real-time," and "Each answer includes numbered citations linking to the original sources."

## Google AI Overviews and AI Mode

- Google says "There are no additional requirements to appear in AI Overviews or AI Mode," "nor other special optimizations necessary."
- To be eligible, "a page must be indexed and eligible to be shown in Google Search with a snippet."
- AI Mode may answer by "issuing multiple related searches across subtopics and data sources," so one question can draw on pages found by several searches.
- And there is no promise: "Indexing and serving isn't guaranteed."

## Claude

- Anthropic separates search from training. "Claude-SearchBot navigates the web to improve search result quality for users."
- "When individuals ask questions to Claude, it may access websites using a Claude-User agent."
- ClaudeBot is the one for training: it is described as "collecting web content that could potentially contribute to their training."
- On answers that use web search, Anthropic says "Every response includes citations, so you can easily verify sources yourself."

## Microsoft Copilot and Bing

- Bing Webmaster Tools has an AI Performance report that shows "how publisher content appears across Microsoft Copilot, AI-generated summaries in Bing, and select partner integrations."
- Bing's advice for AI search is about the basics: "keeping your website crawlable, fresh, and fully indexed is more important than ever."

## Side by side

| AI tool | Crawler that finds pages for answers | Crawler used for training | Do answers link to sources? |
|---|---|---|---|
| ChatGPT search | OAI-SearchBot (ChatGPT-User fetches pages on request) | GPTBot, a separate setting | Not covered on the pages we read |
| Perplexity | PerplexityBot (Perplexity-User fetches pages on request) | Not covered on the pages we read | "numbered citations linking to the original sources" |
| Google AI Overviews and AI Mode | Google Search's own index: a page must be "indexed and eligible to be shown in Google Search with a snippet" | Not covered on the page we read | Not covered on the page we read |
| Claude | Claude-SearchBot (Claude-User fetches pages on request) | ClaudeBot | "Every response includes citations" |
| Microsoft Copilot | Bing's index: keep the site "crawlable, fresh, and fully indexed" | Not covered on the pages we read | Bing reports how publisher content appears in Copilot |

Each company's own documentation, read November 14, 2026. Links at the end of the page.

## What this means for a business

This part is NewsUSA's reading of the documentation above, not the companies' own words.

1. **Check your robots.txt.** Make sure OAI-SearchBot, PerplexityBot and Claude-SearchBot are allowed. Blocking a training crawler such as GPTBot is a separate choice.
2. **Get indexed in Google and in Bing.** Google's AI features draw on Google's index, and Bing's index serves Copilot. Bing Webmaster Tools now reports Copilot appearances.
3. **Answer the question near the top of the page.** Google needs a page to be eligible for a snippet, and a short, direct passage is the easiest thing to quote and link.
4. **Be written about on other sites.** An AI tool can only cite pages that exist and that its search can find. Coverage on other sites gives it more to find. None of these companies says it favors any particular kind of site.

To see what one assistant actually cites, our [test of 200 business questions](https://trynewsusa.com/ai-search-citation-study) logged every site Claude cited with web search on.

## What the documentation does not say

On the pages we read, none of the five explains how it ranks or chooses among the pages it finds, keeps a public list of trusted sites, or promises that any page will be cited. Google says it plainly: "Indexing and serving isn't guaranteed." Anyone who promises you a citation in an AI answer is promising something these companies do not.

## Questions

### How does ChatGPT choose which websites to cite?

OpenAI does not publish a formula. Its crawler documentation says OAI-SearchBot finds the sites shown in ChatGPT's search features, and "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers." Allowing OAI-SearchBot is the one step OpenAI recommends.

### How can my business get cited by Perplexity?

Allow PerplexityBot in your robots.txt, as Perplexity recommends, and publish pages that answer questions people ask. Perplexity searches the web in real time and says "Each answer includes numbered citations linking to the original sources." It does not say how it picks among them.

### Do I need special optimization for Google AI Overviews?

Google says no: "There are no additional requirements to appear in AI Overviews or AI Mode," "nor other special optimizations necessary." A page must be indexed and eligible to show with a snippet.

### Should I block GPTBot?

That is a separate decision from AI search. OpenAI says "Each setting is independent of the others," so a site can block GPTBot, the training crawler, and still allow OAI-SearchBot, which finds pages for ChatGPT search.

### Can NewsUSA guarantee that an AI tool will cite my business?

No, and no one can. None of these companies publishes how it chooses sources, and Google says "Indexing and serving isn't guaranteed." NewsUSA places feature stories on news sites; whether an AI tool cites one is up to that tool.

## How this page was made

Every statement about an AI tool is quoted from its maker's own documentation, help center or blog, read on November 14, 2026, with links below. Two OpenAI pages refused our automated reads and are not used. The section "What this means for a business" is NewsUSA's reading, not the companies' words.

- [OpenAI: Overview of OpenAI crawlers](https://developers.openai.com/api/docs/bots), read November 14, 2026
- [Perplexity: Perplexity crawlers](https://docs.perplexity.ai/guides/bots), read November 14, 2026
- [Perplexity Help Center: How does Perplexity work?](https://www.perplexity.ai/help-center/en/articles/10352895-how-does-perplexity-work), read November 14, 2026
- [Google Search Central: AI features and your website](https://developers.google.com/search/docs/appearance/ai-features), read November 14, 2026
- [Claude Help Center: Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.anthropic.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), read November 14, 2026
- [Claude Help Center: Enable and use web search](https://support.claude.com/en/articles/10684626-enabling-and-using-web-search), read November 14, 2026
- [Bing Webmaster Blog: Introducing AI Performance in Bing Webmaster Tools (public preview)](https://blogs.bing.com/webmaster/2026/2/Introducing-AI-Performance-in-Bing-Webmaster-Tools-Public-Preview/), read November 14, 2026
- [Bing Webmaster Blog: Keeping content discoverable with sitemaps in AI powered search](https://blogs.bing.com/webmaster/2025/7/Keeping-Content-Discoverable-with-Sitemaps-in-AI-Powered-Search/), read November 14, 2026