# Can ChatGPT read your site? Seven checks, in order

> Probably, unless something stops it. Check in this order: search crawlers allowed, page indexed with a snippet, words in the HTML without JavaScript, answers first, evidence on the page, markup and dates that match, mentions elsewhere. An llms.txt file comes last, if at all.

By Dion Zeneli · 28 October 2026 · 8 min read

Source: https://www.theweb0.com/blog/can-chatgpt-read-your-site
Filed under: AI search, Checklist, Websites

---

The order matters because each check depends on the one before it. A crawler that is turned away never reads the page. A page it reads as blank can't be quoted, however good the writing is. So the cheap, mechanical faults go first, and the slow work of being worth quoting comes after.

A word on who is reading. ChatGPT, Claude and Perplexity each send their own search crawler to your site. Google's AI Overviews and AI Mode use the normal Google index, so they see what Googlebot sees.[^google-ai]

## 1. Are the search crawlers allowed in?

Open `yoursite.com/robots.txt` and look for these names.

| Product | Search crawler | If you block it |
| --- | --- | --- |
| ChatGPT search | `OAI-SearchBot` | Your pages "will not be shown in ChatGPT search answers"[^openai] |
| Claude | `Claude-SearchBot` | Anthropic stops indexing your content for search[^anthropic] |
| Perplexity | `PerplexityBot` | Perplexity recommends allowing it to appear in its results[^perplexity] |
| Google AI Overviews and AI Mode | `Googlebot` | You are out of Google Search too |

Each company also runs crawlers with other jobs, and they are separate switches. `GPTBot` collects pages that may be used to train OpenAI's models. Blocking it has no effect on ChatGPT search: OpenAI calls each setting "independent of the others".[^openai] So you can say no to training and yes to being found. A blanket rule that blocks everything with "AI" in its name throws the search crawlers out with the training ones.

Then look past robots.txt. Since 1 July 2025, Cloudflare blocks AI crawlers by default and asks every new domain whether to let them in.[^cloudflare] If your site sits behind a CDN or a firewall, check its bot settings too. A perfect robots.txt does nothing for a request that never reaches it.

Changes are not instant. OpenAI and Perplexity both say a robots.txt edit can take about a day to register.[^openai]

## 2. Can Google index the page and show a snippet?

Google says AI Overviews and AI Mode need no special work. Its one technical rule, for a page to appear as a link in either:

> …a page must be indexed and eligible to be shown in Google Search with a snippet…
>
> [Google Search Central, AI Features and Your Website](https://developers.google.com/search/docs/appearance/ai-features)

So a `noindex` tag, a `nosnippet` tag or a tight `max-snippet` value takes a page out of both. Search Console's URL inspection tells you which applies.

There is one new switch to check. Since 31 August 2026, Search Console has a "Search generative AI" control for every site. It is on by default. If someone has switched it to exclude, "links to your site and your site's content won't appear in Search generative AI features".[^gsc] It takes ten seconds to look.

## 3. Are the words in the HTML before any JavaScript runs?

This is the one I'd bet on. Googlebot runs JavaScript.[^guide] Most of the others don't. When Vercel and MERJ measured AI crawler traffic at the end of 2024, "none of the major AI crawlers currently render JavaScript", and they named OpenAI's, Anthropic's and Perplexity's.[^vercel] If your page builds its text in the browser, ChatGPT's crawler gets an empty shell.

**57%** Of 711 top homepages a scanner could read, the share that showed a crawler without JavaScript a nearly empty page. One day, homepages only, run by a vendor. (Source: [mod_pagespeed, 2026](https://modpagespeed.com/blog/can-ai-read-your-website/))

The test takes a minute. In Chrome, open View Page Source (not Inspect, which shows the page after scripts have run) and search for your headline. Or from a terminal:

```sh title="Is the headline in the raw HTML?"
curl -s https://yoursite.com/pricing | grep -i "your headline here"
```

If the words are there, a crawler can read them. If you find an empty `<div id="root">` and a pile of script tags, it can't. The fix is server rendering or static generation, which every current framework offers. Crawlers may get better at JavaScript. I would still build for the raw HTML, because getting it right costs nothing extra on a new build.

## 4. Does each page answer its question first?

Once a page can be read, the question is whether a sentence on it can be lifted out and repeated. Animalz puts the advice plainly: "Answer the question in the first sentence clearly and completely."[^animalz] Google says there is "no requirement to break your content into tiny pieces" for AI.[^guide] Both are right. You don't need to chop the page up. You do need the answer near the top, because the buyer wants it there too.

**Hard to quote**

Helping ambitious teams build what comes next.

**Easy to quote**

Bookkeeping software for dental practices in Belgium. From 49 euros a month.

Both lines are invented for this note. Only the second can be repeated as an answer to anything. The same goes for every page that matters: the pricing page says the price, the services page says what you do and for whom, in the first two lines.

## 5. Is there evidence a machine can quote?

In a lab test published at KDD 2024, the authors rewrote web pages in nine ways and measured how much a generative engine used each version in its answers. Three changes did best: citing sources, adding quotations and adding statistics.

**30–40%** The relative gain in visibility in AI answers when pages cited sources, added quotations or added statistics. Keyword stuffing gave little to no gain. (Source: [Aggarwal et al., GEO, KDD, 2024](https://arxiv.org/html/2311.09735v3))

Read it for the direction, not the size. The engine was built on GPT-3.5 and the test is from 2023 and 2024. Google's advice points the same way. It asks for "unique, non-commodity content" that people find helpful.[^mueller] The cheapest evidence you have is your own: your prices, your process, numbers from your own work, with the source next to each one.

## 6. Do the markup and the dates match what people see?

Structured data is optional. Google says it "isn't required for generative AI search" and that no special schema is needed.[^guide] If you use it, Google's rule is that "all the content in your markup is also visible on your web page".[^mueller] Markup that says something the page doesn't is worse than none.

Dates are simpler. Google asks you to "Add a user-visible date to the page and feature it prominently."[^dates] An article with no date and no author gives a reader, or a machine, no way to judge how fresh it is or who stands behind it.

## 7. Does anyone else mention you?

This is the slow one. Ranking well helps, but it no longer decides much on its own. Google says its AI features run a "query fan-out": several related searches at once, across subtopics.[^google-ai] Ahrefs then found that most pages cited in AI Overviews were not in the first ten results for the search that was typed.

*Drawing: Where the pages cited in Google's AI Overviews ranked for the original search, counting every block on the results page. Counting blue links only gave 37.1%, 26.2% and 36.7%. Source: [Ahrefs, March 2026](https://ahrefs.com/blog/ai-overview-citations-top-10/). See it at https://www.theweb0.com/blog/can-chatgpt-read-your-site*

What does go with being cited is being talked about. In an Ahrefs study of 75,000 brands, mentions of a brand across the web tracked visibility in AI Overviews far more closely than links did.

**0.664** Correlation between brand mentions on the web and brand visibility in AI Overviews. (Source: [Ahrefs, 75,000 brands, 2025](https://ahrefs.com/blog/ai-overview-brand-correlation/))

**0.218** Correlation for backlinks. Same study. (Source: [Ahrefs, 75,000 brands, 2025](https://ahrefs.com/blog/ai-overview-brand-correlation/))

That is a correlation, and big brands get both. It doesn't mean you should go and buy mentions: Google says that "seeking inauthentic 'mentions' across the web isn't as helpful as it might seem".[^guide] The real version is slower. A client's case study that names you, a talk, a podcast, a comparison page someone else wrote. No developer can fix this one in an afternoon, which is exactly why it comes last.

## What can you skip?

> **llms.txt.**
>
> An `llms.txt` file is a proposed markdown file that tells AI agents what is on your site and where.[^llms] In a study of nearly 300,000 domains, about one in ten had one, and the authors found "no correlation" between having it and being cited.[^seranking] Google says it will "neither harm nor help" in Google Search, which ignores it.[^guide] Add one if it takes ten minutes. Don't let it jump the queue.

Three more things you can leave alone. Splitting pages into tiny chunks: Google says it isn't needed. A separate page for every way someone might phrase a question: Google says doing that to manipulate rankings or AI answers breaks its spam policy on scaled content.[^guide] And any tool that sells you an "AI visibility score" built on insider data: Google says no outside tool has access to its ranking or AI systems.[^guide]

## The seven checks, in one list

Hand this to whoever looks after the site.

- robots.txt allows `OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot` and `Googlebot`, whatever it says about training crawlers.
- The CDN or firewall lets those crawlers through.
- The pages that matter are indexed and allowed a snippet. The Search Console AI control is set to include.
- The headline and main text are in the page source, without JavaScript.
- Each page answers its question in the first two lines.
- Numbers on the page have a source, and quotations have a name.
- Structured data matches the visible text. Articles show a date and an author.
- Someone other than you has a reason to mention you.

That is eight lines for seven checks, because the first one has two doors.

A tool can run the mechanical half of this list. It can't judge whether your answers are any good, and it can't see who mentions you.

To run the first three checks on your own page, use the [site check](https://www.theweb0.com/tools/site-check).

[^google-ai]: Google: "There are no additional requirements to appear in AI Overviews or AI Mode", and both "may use a 'query fan-out' technique". [Google Search Central, AI Features and Your Website](https://developers.google.com/search/docs/appearance/ai-features), checked 11 November 2026.

[^openai]: OpenAI's crawler documentation, including the note that it "can take ~24 hours" after a robots.txt change. [OpenAI, Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots), checked 11 November 2026. Perplexity says "up to 24 hours". [Perplexity Crawlers](https://docs.perplexity.ai/docs/resources/perplexity-crawlers).

[^anthropic]: Anthropic lists three robots: ClaudeBot for training, Claude-User for fetches a user asks for, and Claude-SearchBot for search. [Claude Help Center](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), checked 11 November 2026.

[^perplexity]: Perplexity says PerplexityBot "is not used to crawl content for AI foundation models". [Perplexity Crawlers](https://docs.perplexity.ai/docs/resources/perplexity-crawlers), checked 11 November 2026.

[^cloudflare]: Cloudflare's press release of 1 July 2025: AI crawlers are blocked "by default", and every new domain is asked at sign-up whether to allow them. [Cloudflare press release](https://www.cloudflare.com/press/press-releases/2025/cloudflare-just-changed-how-ai-crawlers-scrape-the-internet-at-large/), checked 11 November 2026.

[^gsc]: Google: "This is the default control for all properties." The note at the top says it rolled out to all sites on 31 August 2026. [Search Console Help, Search generative AI control](https://support.google.com/webmasters/answer/16908024), checked 11 November 2026.

[^vercel]: The study counted GPTBot, ClaudeBot, PerplexityBot and others across Vercel's network in late 2024. Gemini uses Googlebot's renderer, and Applebot renders too. [Vercel, The rise of the AI crawler](https://vercel.com/blog/the-rise-of-the-ai-crawler), 17 December 2024.

[^animalz]: [Animalz, 20 Techniques That Get You Cited in Answer Engines](https://www.animalz.co/blog/ai-aeo-answer-engine-citation), last updated 1 May 2026.

[^guide]: All from one page, last updated 10 July 2026: [Google's Guide to Optimizing for Generative AI Features on Google Search](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide), checked 11 November 2026.

[^mueller]: [Google Search Central Blog, Top ways to ensure your content performs well in Google's AI experiences on Search](https://developers.google.com/search/blog/2025/05/succeeding-in-ai-search), 21 May 2025.

[^dates]: [Google Search Central, Influence your byline dates in Google Search](https://developers.google.com/search/docs/appearance/publication-dates), checked 11 November 2026.

[^seranking]: Just 10.13% of the domains had the file. [SE Ranking, LLMs.txt: Why Brands Rely On It and Why It Doesn't Work](https://seranking.com/blog/llms-txt/), checked 11 November 2026.

[^llms]: The proposal, by Jeremy Howard, last changed 10 August 2026. [llmstxt.org](https://llmstxt.org/), checked 11 November 2026.