Organiq

Is my site blocking ChatGPT? Check which AI crawlers can access it

· Organiq · third-party facts carry source and date in the text

ChatGPT search leaves out a page when your robots.txt blocks OAI-SearchBot for it; blocking GPTBot only opts the content out of OpenAI model training. Paste a URL below: the check reads robots.txt for that exact path for 8 AI crawlers, plus llms.txt and the page's JSON-LD. No sign-in; Organiq does not save the URL.

Reads the page, /robots.txt and /llms.txt of the site once, from Organiq's server. No sign-in; Organiq does not save the address.

What does each AI crawler in the check do?

The 8 in the check: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, Bingbot, CCBot and Applebot-Extended. Official sources read on November 3, 2026:

Google-Extended and Applebot-Extended are usage control tokens, not crawlers: they only matter inside robots.txt and never show up in server logs.

What does robots.txt not control?

How do I read the result?

Each crawler gets one of four states for the path of the URL you pasted:

Matching rules (RFC 9309 and Google): a crawler follows only the group that names it; without one, it follows *. Among the rules that match the path, the longest wins; a tie between Allow and Disallow goes to Allow. robots.txt with 404 or another 4xx: crawlers may read everything. With 5xx: crawlers must treat the whole site as disallowed while the error lasts.

How do I allow or block one AI crawler?

In /robots.txt at the site root, one group per crawler. This example keeps the page in ChatGPT search and out of OpenAI training:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

Once a crawler has its own group, it stops following the * group: repeat there the * rules that should still apply. Blocking training crawlers (GPTBot, CCBot, Google-Extended, Applebot-Extended) is a choice; it does not take the page out of search.

What should llms.txt contain?

llms.txt is a proposal published at llmstxt.org: a Markdown file at /llms.txt. The only required item is a # Name title; then a blockquote summary (>) and ## sections with lists of links. The check counts a file as valid when it answers 200, is not an HTML page and starts with # Title, and it counts the links. It does not tell whether any AI reads the file.

# Example Co

> One sentence on what the site is and who it is for.

## Docs

- [Pricing](https://example.com/pricing): plans and prices
- [Guide](https://example.com/guide): how to start

What does the check look for in JSON-LD?

It reads the <script type="application/ld+json"> blocks in the raw HTML (JSON-LD added by JavaScript after load is not seen) and lists the types. On a content page it looks for a citable type (Article, BlogPosting, NewsArticle, TechArticle, FAQPage or HowTo) with a Person author that has a url, datePublished and dateModified, and an Organization with sameAs: that says who wrote it, when, and who publishes it. On a home page, Organization alone is expected. Minimal structure of an article:

{
  "@context": "https://schema.org",
  "@type": "Article",
  "headline": "How to choose a CRM",
  "datePublished": "2026-10-01",
  "dateModified": "2026-11-03",
  "author": { "@type": "Person", "name": "Ana Lima", "url": "https://example.com/ana" },
  "publisher": { "@type": "Organization", "name": "Example Co", "url": "https://example.com",
    "sameAs": ["https://www.linkedin.com/company/example"] }
}

What differs from other free AI crawler checkers, and what is the exception?

Frequently asked questions

If I block GPTBot, does my site disappear from ChatGPT?

No. OpenAI documents GPTBot for model training and OAI-SearchBot for ChatGPT search; each robots.txt setting is independent. To stay out of training and in search, block GPTBot and allow OAI-SearchBot.

Does blocking Google-Extended remove my site from Google Search?

No. Google says Google-Extended does not affect inclusion or ranking in Google Search. It only decides whether content Google crawls may train and ground Gemini models.

robots.txt allows a crawler, so why can it still not read my pages?

robots.txt is a request, not a lock. A firewall, a CDN bot rule or a login can stop the crawler even when robots.txt allows it. This check reads the files; it does not send requests as each crawler.

What if my site has no robots.txt?

When robots.txt answers 404 or another 4xx, crawlers may read everything (RFC 9309), so all 8 show as allowed with no rule. When it answers 5xx, crawlers must treat the whole site as disallowed until it answers again; the check shows "could not tell".

Does Organiq save the URL I check?

No. The check runs when you submit and the result exists only on that page; the URL is not written to Organiq's database, and Google Analytics on the result page receives the address without the URL you typed. The usage limit keeps your IP in the server's memory only, never written to a database or file, and drops it after one minute.

Is the check free? Is there a limit?

It is free and needs no sign-in. The limit is about 8 checks per minute per connection.

Does allowing AI crawlers make AI answers cite my site?

There is no guarantee. Allowing a crawler only means it may read the page; citation depends on the question and the content. Search Console shows the Google searches where the page appears.

Which searches bring people to the site?

robots.txt says who may read a page; it does not say which searches lead people to it. Google Search Console does, for sites you have verified: impressions, clicks, CTR and average position for each query and each page. Guide: connect Search Console to Organiq.

The same checks on the page open in Chrome: the free Organiq GEO extension.

See also

Written by , founder of Organiq · Updated