We Tested 46 SEO and AI Visibility Vendors: Do They Let AI Crawlers In?

On 22 August 2026 we sent ten AI crawler user agents at 46 SEO platform and AI visibility vendor websites and recorded what came back. The premise was simple: these are the companies selling advice about AI search visibility. Do their own sites let the AI crawlers in? Mostly yes — but the exceptions are instructive, and the way most of them document their policy is broken in both directions.

What did the test actually find?

Of 46 domains tested, 43 returned a normal response to a baseline browser request and form the working sample. Three were excluded because they blocked our requests regardless of user agent, which tells us about their bot policy generally rather than their AI crawler policy specifically.

We want to be careful about how much weight that carries. Fourteen AI visibility tools is a small sample, and three blockers out of fourteen is not a trend — it is three companies. What makes profound.ai notable is not the percentage but the specificity: this is a deliberate, user-agent-keyed block on the exact crawlers that feed the assistants the product measures.

How did we test this?

For each domain we fetched robots.txt once, then issued one request to the homepage per crawler user agent, following redirects, with a 20-second timeout and a short delay between requests. We recorded the HTTP status for each. A status of 401, 403, 429, a timeout, or a connection error counts as a block; anything under 400 counts as access.

The ten user agents were GPTBot, OAI-SearchBot and ChatGPT-User (OpenAI), ClaudeBot (Anthropic), PerplexityBot (Perplexity), Google-Extended (Gemini training), Applebot-Extended (Apple), CCBot (Common Crawl, which feeds many training sets), meta-externalagent (Meta), and Bytespider (ByteDance).

Which vendors block which crawlers?

Six domains blocked at least one crawler. They are not equivalent cases, and lumping them together would be misleading — so here they are individually.

Strip out Bytespider — which many sites block for bandwidth reasons and which does not feed a major consumer assistant — and strip out marketmuse's rate limiting, and the picture narrows sharply. Two of 43 vendors deliberately block the crawlers behind ChatGPT, Claude or Perplexity: profound.ai and majestic.com. That is the honest version of the finding.

Blocking these crawlers is a legitimate business decision. A company may not want its content in training corpora, and there is a real argument that assistants answering from your content without sending traffic is a bad trade. The tension is only worth naming when the same company sells visibility inside those assistants.

Why can't you trust robots.txt?

This is the finding with the widest application, and it cuts both ways. In this sample robots.txt was an unreliable guide to actual crawler access in both directions.

Every live block we found was undeclared. All 14 domain-and-crawler pairs that returned a block had a robots.txt that permitted that crawler. The blocking happens at the CDN or WAF layer, invisible to anyone auditing policy by reading robots.txt.

And declared blocks were not enforced. brightlocal.com and knowatoa.com both disallow seven AI crawlers in robots.txt — GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, meta-externalagent and Bytespider — and then served every one of them a 200. knowatoa.com is itself an AI visibility tool. A compliant crawler honours the directive and leaves; a non-compliant one walks straight in. The file states an intent the server does not enforce.

There is a third gap worth noting: only 11 of 43 vendors — 26% — name any AI crawler in robots.txt at all. The other 74% have made no explicit decision either way. Their AI crawler policy is whatever their wildcard rule and their CDN's bot-mitigation defaults happen to produce, which is not the same thing as a policy.

What did we get wrong the first time?

We are including this because it changes results and we expect other people running this test to hit it.

Our first pass used HTTP HEAD requests, which are cheaper and are the obvious choice when you only want a status code. HEAD produced false positives. semrush.com, moz.com and similarweb.com all appeared to block crawlers under HEAD and all return 200 to an identical GET. They reject the HEAD method, not the crawler. Publishing the HEAD numbers would have accused three named vendors of something they do not do.

A second correction came from re-testing. marketmuse.com looked like a blocker on the first pass and returned 429 on the second — that is rate limiting under repeated requests, not a user-agent block. Any single-pass crawler study will misclassify rate limiting as blocking.

How should you check your own site?

The test is trivial to run against a single domain and takes about a minute. For each crawler you care about, request your own homepage with that user agent and read the status:

• •curl -s -o /dev/null -w "%{http_code}" -A "GPTBot/1.1" https://yourdomain.com
• •Repeat for OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and Google-Extended.
• •Run the same request with an ordinary browser user agent as a control.
• •Anything that returns 403, 401 or 429 to a crawler but 200 to the browser is a block you may not know you have.

The most common cause of an unintended block is not robots.txt — it is a bot-mitigation setting at the CDN. Cloudflare's AI crawler controls in particular can block GPTBot by default on some plan configurations, which means a site can acquire an AI crawler block without anyone editing a file. If your test shows a block you did not intend, check the CDN before the repository.

Then decide deliberately. Allowing these crawlers is what makes citation possible; blocking them is a defensible choice about training data and traffic. The failure mode this study documents is neither of those — it is having a policy by accident, and a robots.txt that describes a different policy than the server enforces.

Where is the full dataset?

The complete result set covers 46 domains × 10 crawler user agents plus a browser control and a robots.txt policy parse for each — every status code behind every number above. Both the raw data and the script that produced it are published: download the full dataset (JSON) or read the test script. It has no dependencies beyond Node, so the study is reproducible end to end.

Vendors: if this test misrepresents your configuration — particularly if you believe an IP-reputation rule rather than a user-agent rule produced your result — we will re-test and correct the record. Accuracy matters more to us than the finding.