Journal
How to check if AI crawlers can access your website
To check if AI crawlers can access your website, read your live robots.txt for rules naming GPTBot, OAI-SearchBot, PerplexityBot and ClaudeBot, then fetch real pages and confirm the response codes. Permission is not proof, so finish in your CDN, firewall and server logs.
Table of Contents
- Why allowed is not the same as accessible
- Check your robots file for GPTBot, OAI-SearchBot, PerplexityBot and ClaudeBot
- How to check if AI crawlers can access your website at the page level
- Review Cloudflare and CDN bot settings that drop AI crawlers
- How to check if AI crawlers can access your website in server logs
- Test your crawler access with the free Findability Check
- Frequently Asked Questions
Why allowed is not the same as accessible
AI answers can't cite you if your server turns away the bots that read your pages. Most advice on this stops at the robots file. That file states permission. It doesn't prove a single request ever succeeded.
Two terms carry the rest of this article.
robots.txt: a plain-text, UTF-8 file at the root of your host that tells compliant crawlers which URL paths they may request. It controls crawling. It doesn't control indexing or whether you appear in an AI answer.
Crawler access: whether a named bot can request a real page on your site and get the actual content back, with no login, challenge page, block or server error in the way.
The gap between those two is where sites go quiet. A server can block a crawler that robots.txt allows. Firewall rules, CDN bot management, WAF policies, IP deny lists, login walls, rate limits and errors such as 403, 429 or 5xx all do it.
Don't treat an Allow rule as proof that GPTBot or OAI-SearchBot reached your pages. The bot reads your robots file, but the block happens at your CDN, firewall or origin server. The robots file never finds out.
To check AI crawler access: (1) read your live robots.txt for named bots, (2) fetch a real page and read the status code, (3) find that bot's requests in your logs.
One more thing before the steps. You'll see percentages online claiming how many sites block AI bots. The official documentation from OpenAI and Google gives no such universal figure. Any number like that needs a defined sample, date, crawler list and test method. Without those, treat it as unverified and test your own site instead.
Here is what each check can and can't tell you:
| Check | What it proves | What it can't prove | Where you run it |
|---|---|---|---|
| Read /robots.txt | Which paths a compliant bot may request | That any request succeeded, or that non-compliant bots stay out | Browser, at the root of each host |
| curl a real page | The status, redirects and headers a public request receives | How your CDN treats the real bot's IP address | Terminal |
| CDN, WAF or firewall events | Whether a crawler was blocked, challenged or rate-limited at the edge | What your origin would have returned | CDN or security dashboard |
| Origin server logs | Timestamp, user-agent, IP, URL, status and size of each hit | Anything stopped before it reached the origin | Hosting panel or log files |
If you want an outside read before you open a terminal, our free Findability Check needs no account. The last section covers what it can and can't see.
Check your robots file for GPTBot, OAI-SearchBot, PerplexityBot and ClaudeBot
Open https://yourdomain.com/robots.txt in a browser. Read the live file the server returns, not the copy in your repo or CMS settings. What the server sends is what bots read.
Search the file for groups naming these crawlers:
OAI-SearchBotGPTBotClaudeBotPerplexityBotGoogle-ExtendedGooglebot
Then look for the broad rule that catches people out:
User-agent: *
Disallow: /
That blocks every crawler covered by the wildcard. The Google robots.txt specification explains the exception: a more specific group wins. If your file also has a User-agent: GPTBot group, GPTBot follows that group instead of the wildcard.
Next, test the path. Disallow: / blocks the whole site for that crawler. Disallow: /private/ blocks only URLs beginning with /private/. Check the exact paths you care about, such as /blog/ or /products/, against each rule.
OpenAI runs two bots, and they're separate switches
As of October 2026, OpenAI's crawler documentation splits its bots by job. OAI-SearchBot is used for search features. GPTBot crawls content that may be used to improve OpenAI's foundation models. In OpenAI's words, "Each setting is independent of the others."
So blocking GPTBot doesn't necessarily block OAI-SearchBot. If you want to appear in ChatGPT search but keep your content out of model training, the file looks like this:
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
OpenAI also notes that when both bots are allowed, one crawl may sometimes serve both purposes. Check the syntax before you deploy. The file must be plain text, UTF-8, with valid line formatting.
OpenAI search systems can take about 24 hours to reflect a robots.txt change (OpenAI crawler documentation, 2026). Don't judge a fix the same afternoon.
How to check if AI crawlers can access your website at the page level
A successful request for robots.txt proves one thing: the robots file loads. It says nothing about your actual pages. Test a handful of representative URLs, such as the homepage, a blog post, a service or product page and one deep page.
Start with a header request:
curl -I https://yourdomain.com/page
Read four things in the response:
- Status code. You want a success. A 403, 429 or 5xx is a problem.
- Redirects. Follow every
Locationheader until you reach the final URL. - Authentication challenges. Any prompt for credentials means a crawler can't get in.
- Security headers. Note anything your CDN or firewall adds, since it hints at which layer answered.
curl -I sends a HEAD request, and some setups answer HEAD differently from a normal page load. Run a GET as well and print the status plus the size of what came back:
curl -s -o /dev/null -w "%{http_code} %{size_download}\n" https://yourdomain.com/page
Size matters. A real article and a challenge page can both return a success code, but they won't weigh the same.
You can also send a crawler's name as the user-agent:
curl -I -A "GPTBot" https://yourdomain.com/page
This shows whether a rule keyed to the user-agent string treats that name differently. It has a hard limit. If your CDN verifies bots by IP address, your laptop isn't the bot. A block here may not mean the real crawler is blocked, and a pass doesn't mean it's allowed. Logs settle that question.
Finally, confirm the page is public. Open it in a private window with no cookies, no login and no special headers. If content sits behind a login, a JavaScript challenge or a cookie wall, a crawler may be permitted by robots.txt and still be unable to retrieve or render the page.
Review Cloudflare and CDN bot settings that drop AI crawlers
Your CDN sits in front of your server. It decides who gets through before your robots file matters at all. If you use Cloudflare or any other CDN, this is the layer to check next.
Review these settings:
- Bot management. Look at how automated traffic is classified and which action applies to each class. Check whether AI crawlers land in a group set to block or challenge.
- WAF custom rules. Search for rules that match user-agent strings. A rule written to stop scrapers by matching words like "bot" can catch the crawlers you want.
- IP deny lists. Lists copied from forums or older setups sometimes include ranges you didn't mean to block.
- Rate limits. A crawler fetching many pages quickly can trip a limit and start getting 429 responses.
- Challenge pages. A JavaScript challenge stops a person for a second. It can stop a crawler completely.
The same applies on your origin. Server firewalls and security plugins can block crawlers with no help from the CDN.
Then open your CDN's security event log. Filter for user-agents containing GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot. For each event, note the action taken and the reason given.
One rule keeps this tidy: make your layers agree. If robots.txt allows OAI-SearchBot and blocks GPTBot, your edge rules should say the same thing. When they disagree, the edge wins and nobody sees it happen.
For the exact location of each control, use Cloudflare's own documentation. This article doesn't map their menus, because the setting names are theirs to change.
How to check if AI crawlers can access your website in server logs
Logs are the most reliable evidence you have. Robots rules and curl tests tell you what should happen. Server, CDN and WAF logs tell you what did.
For each crawler request, confirm:
- Request timestamp
- User-agent
- Source IP
- Requested URL
- HTTP status
- Response size
- Block or challenge reason, from the CDN or WAF
Check both layers. Your origin logs only show requests that got past the edge. If a crawler never appears in origin logs, the CDN log is where you'll find out why.
Use this table to read what you find:
| What you see | Likely cause | Where to look next |
|---|---|---|
| No requests from the crawler at all | Blocked at the edge before the origin, or not crawled yet | CDN or WAF event log |
| 403 | Firewall rule, WAF policy or IP deny list | Security rules matching that user-agent or IP |
| 429 | Rate limit | CDN or server rate-limit settings |
| 5xx | Server error during the crawl | Origin error logs, hosting status |
| Chain of 301 or 302 | Redirects between hosts or protocols | Redirect rules and canonical host settings |
| Success code with a tiny response size | Challenge page or empty shell served instead of content | Bot challenge settings, JavaScript rendering |
| Login page or credential prompt | Authentication required on that path | Access settings for that section |
Make sure the bot is the bot
User-agent strings can be forged. Anyone can send "GPTBot" from a laptop, as the curl test above shows. A request that claims to be an AI crawler isn't proof that it is one.
Verify with provider-published IP ranges, and reverse DNS where the operator supports it. Google is the clearest example. It publishes crawler user-agents and IP ranges, and its common crawlers documentation explains how to confirm a suspected Googlebot request. The same page states that Google's common crawlers "always obey robots.txt rules when crawling automatically."
For other operators, look up whether they publish IP ranges before you trust or block by name. If they don't, judge requests by behavior and treat the user-agent as a claim, not an identity.
Test your crawler access with the free Findability Check
External checks see your site the way a stranger does. That makes them a quick first pass. You don't need log access, a terminal or a developer to run one.
They also have limits you should know. An outside tool can't read your CDN event log or your origin logs. It doesn't request pages from the real crawler's IP address. So it can flag an obvious problem, but it can't prove a specific bot got through. Pair it with the log check above.
ADpilot's free Findability Check gives you that outside read with no account. It doesn't read your server logs, open your Cloudflare dashboard or edit your robots file. Those fixes stay with you or your developer. Inside the paid product, ADpilot measures Google, Bing and AI answers in one place, so you can see whether fixed access turns into AI search visibility over time, tracked in ADpilot reports.
Crawler access isn't a one-time setting. Rerun all three checks (robots file, page request, logs) whenever one of these happens:
- You edit robots.txt. Wait out OpenAI's adjustment window before you judge the result.
- You change CDN, WAF or firewall rules.
- You install or update a security plugin.
- You migrate hosts, change domains, or switch between
wwwand non-www, orhttpandhttps. - You put a new section of the site behind a login or cookie wall.
Write down what you found each time, including date, crawler, URL, status and fix. When something breaks later in 2026, that record tells you which layer changed.
Frequently Asked Questions
Does a noindex tag stop AI crawlers from reading a page?
No. noindex is an indexing directive, not an access control, and a crawler generally has to retrieve the page to see it. The reverse also holds: Google Search Central says robots.txt is "not a mechanism for keeping a web page out of Google."
Does one robots.txt cover both my www and non-www site?
No. Robots rules are specific to protocol and host, so a file on https://www.example.com doesn't govern http://example.com or any other subdomain. When you work out how to check if AI crawlers can access your website, open the robots file on every host and protocol that serves pages.
Can I put robots.txt in a subfolder like /blog/?
No. The file must sit at the top-level path, such as https://example.com/robots.txt. A file at /blog/robots.txt doesn't control the site, and crawlers won't treat it as your robots file.
Do all AI crawlers have to obey robots.txt?
No verified industry-wide rule requires every AI crawler to obey it. Robots.txt is a voluntary protocol. That's why any method for how to check if AI crawlers can access your website has to end in your server and security logs, not in the robots file.
Do Google's special-case crawlers follow my robots.txt rules?
Not always. Google's special-case crawlers documentation notes that some of these crawlers may operate under separate agreements and may not follow ordinary global robots.txt rules. Check that page for the crawler you care about before writing rules for it.
Conclusion
Knowing how to check if AI crawlers can access your website comes down to three layers: the robots file, a real page request and your logs. The robots file tells you what's permitted. Curl tells you what a public request receives. Your CDN and origin logs tell you what actually happened to GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot, so fix whichever layer disagrees with the others.
Start your 7-day free trial of Scale, no card: adpilot.ee/register
