AI Crawler Blocker:
Block AI Bots by IP, Not Just User-Agent
Pick the crawlers you want gone and this AI crawler blocker writes four working configurations: a robots.txt block, an nginx rule, an Apache .htaccess rule, and a Cloudflare WAF expression. The firewall rules are built from each operator's own published CIDR ranges, read live rather than pasted from a list that went stale months ago. Every crawler shows what blocking it costs you, and you can check any address from your logs before you commit to a rule.
Quick Answer: How Do You Block AI Crawlers?
To block AI crawlers, add a robots.txt rule naming each bot, then enforce it with a firewall rule that matches the operator's published IP ranges. robots.txt is a request a crawler can ignore. An IP-range block is the only layer a spoofed user-agent cannot walk past.
Verify a crawler IP before you block it
Take an address straight from your access log. This checks it against every published crawler range on this page, then runs forward-confirmed reverse DNS — the check that separates a real crawler from something wearing its name.
IPv4 or IPv6. Nothing is stored or logged. Private and reserved addresses are rejected because a public crawler cannot use one.
Keep going with this address
Each of these opens with already filled in.
Choose what to block
Bots are grouped by what they do with your content. Blocking a whole group is rarely the right answer — read the cost line under each one.
Ranges refresh from each operator's own feed at most every six hours. 7 of 7 feeds resolved on this request. Nothing you select here is stored or logged — the rules are assembled in your browser.
Jessica Wright
Cybersecurity Threat Researcher
Jessica works on bot verification and IP reputation, and spends most of her time separating traffic that says what it is from traffic that only claims to.
The mistake I see most often is a site that blocks GPTBot, congratulates itself, and keeps getting crawled. The user-agent header is a sentence the client writes about itself. Nothing verifies it. When I pull a week of logs for a site that has "blocked AI", a large share of what remains is either a bot with no declared identity at all or a declared one arriving from an address the operator never published. That is why the first box on this page is a lookup rather than a checkbox.
One limit worth stating plainly: IP ranges move. As this page loaded, OpenAI's training list carried a October 2025 stamp, its search list January 2026, Perplexity's February 2025 and Anthropic's August 2026. A rule built from any of them is accurate on the day you paste it and decays from there. Automate the refresh, because a stale list fails in the direction of letting traffic through.
Last reviewed 26 August 2026 · Ranges read directly from operator-published feeds · No queries logged
View all articles by Jessica WrightWhat this AI crawler blocker actually generates
This tool writes four separate configurations from one set of checkboxes, because blocking AI crawlers properly takes two layers, not one. The first layer is robots.txt, a plain-text file at your site root that names each crawler and asks it to stay out. The second layer is a firewall rule that matches the request's source IP against the addresses the operator has publicly said its crawler uses.
The difference between those two layers is the difference between a sign and a lock. RFC 9309, the IETF standard that finally formalised robots.txt in 2022, says so in its own security section: the Robots Exclusion Protocol is not a substitute for real content security measures. It is a published preference. Compliance is voluntary, and it always has been.
So the generator gives you both:
- robots.txt — the polite, standards-compliant request. This is what well-behaved crawlers read, and for the majority of documented AI bots it is genuinely sufficient.
- nginx — a
geoblock listing the published CIDR ranges, plus a user-agent fallback for operators who publish no ranges. - Apache .htaccess — the same logic in
mod_authz_coresyntax, for shared hosting where you do not control the server config. - Cloudflare — a WAF expression you can paste straight into a custom rule.
Read the cost line. Every crawler in the picker carries a one-line note about what you lose by blocking it. Training bots cost you nothing in visibility. Search bots cost you citations in ChatGPT, Claude and Perplexity answers. Agent bots cost a real person, right now, the page they asked for. Those are three different decisions and most generators bundle them into one button.
Why blocking by user-agent is not really blocking
A user-agent is a header the client writes about itself. Nothing in HTTP verifies it. Anyone can send a request that says GPTBot/1.4 from any machine on the internet, and a rule that matches on that string will treat it exactly like the real thing. This cuts both ways, and the second direction is the one that catches people out.
If you allow a bot by user-agent, a scraper types that string and walks in. If you block a bot by user-agent, a scraper drops the string entirely and walks in as a generic browser. Either way the header is doing no work. This is the same class of problem as forged source addresses in IP spoofing: a claim is not evidence.
What the operators actually give you
The serious operators know this, which is why most of them publish a machine-readable list of the IP ranges their crawlers run from. Matching the request IP against that list is a check the client cannot influence, because the address comes from the TCP connection rather than from anything the client typed. Some go further and support FCrDNS — forward-confirmed reverse DNS, where you resolve the IP to a hostname, resolve that hostname back to an IP, and confirm you land on the address you started with. That is the second half of what the verification box at the top of this page does, and it is worth pairing with a plain reverse IP lookup on the address when you want to see what else lives on the same host.
Common Crawl's feed explicitly recommends FCrDNS on top of its range match. Amazon documents a reverse-DNS scheme under crawl.amazonbot.amazon. Others publish only ranges. And several publish nothing at all, which is a meaningful fact about those crawlers rather than an oversight you can work around.
Worth knowing: matching an IP to a range tells you the request came from that network. It does not tell you what the request will do with your content. Those are different questions, and only the first one is answerable from an address. Use our ASN lookup to see which network operator an address belongs to when a request matches no published list, and check whether it is simply a rented datacenter address before assuming anything worse.
Training, Search, Agent: the split that changed everything in 2026
Until recently the question was binary: block AI bots, or do not. That framing broke down once the same companies started running several crawlers for genuinely different jobs. On 1 July 2026 Cloudflare replaced its single toggle with three categories, and the taxonomy is now the clearest way to think about the decision regardless of whether you use Cloudflare at all.
- Training — the crawler takes your content to train or fine-tune a model. GPTBot, ClaudeBot, CCBot, Bytespider.
- Search — the crawler indexes your content so an engine can answer questions about it later, with a citation. OAI-SearchBot, PerplexityBot, Claude-SearchBot.
- Agent — the crawler fetches a page in real time because a person asked for it. ChatGPT-User, Perplexity-User, Claude-User.
Blocking training is a content-rights decision with no visibility cost. Blocking search removes you from AI answers entirely, which is almost never what a site actually wants. Blocking agents means a human being gets a worse answer about your own business, in the moment they asked.
OpenAI is the clearest example of why the split matters. It runs four documented crawlers doing four different jobs, each with its own robots.txt token, and OpenAI's crawler documentation keeps them deliberately separate so you can allow one and refuse another. Treating "OpenAI" as a single thing to block is the most common configuration error on this topic.
How to opt out of AI training without losing search visibility
This is what most people actually want, and it has a clean answer. Opting out of AI training does not require blocking anything that affects search — you just have to use the training-specific tokens and leave the rest alone.
- Block
GPTBot,ClaudeBot,CCBotandBytespider. These only feed model training. - Add
Google-ExtendedandApplebot-Extended. These opt you out of Gemini and Apple Intelligence training while Googlebot and Applebot keep indexing you normally. - Leave
OAI-SearchBot,PerplexityBotandClaude-SearchBotallowed, so you stay citable in AI answers. - Leave the agent crawlers allowed too, unless you genuinely want a person asking about your site to get an answer that does not include it.
The Training only preset in the generator above builds exactly this. One caveat applies at the IP layer rather than the robots.txt layer: GPTBot and OAI-SearchBot publish overlapping prefixes, so a firewall rule aimed at training will clip search on those addresses. If keeping ChatGPT search is the priority, use robots.txt for OpenAI and save the firewall for crawlers that ignore it.
The 15 September 2026 default, and the trap inside it
Cloudflare has set new defaults that take effect on 15 September 2026. Training and Agent crawlers will be blocked by default on pages that display ads, while Search stays allowed. Per Cloudflare's own announcement, the new defaults apply to newly onboarded domains, new sites added to existing accounts, and free-tier customers, and site owners can opt out in their zone security settings before the date.
The part that is easy to miss matters more than the headline. Cloudflare enforces the most restrictive applicable rule, and multi-purpose crawlers get judged on all of their behaviours. Googlebot, Applebot and BingBot each do search and feed AI products, and Cloudflare names all three explicitly. So if you tell Cloudflare to block Training, those three get blocked too — including the crawler your Google rankings depend on.
The legacy toggle is the real trap. Cloudflare's announcement is specific that this applies to customers who selected to block Training either through the new AI traffic options or through the older Block AI bots service. That older toggle has existed for years. Plenty of sites switched it on once, filed it under done, and have not looked at it since.
If that describes you, nothing on your end changes on 15 September — but Googlebot can start getting blocked on your ad-serving pages anyway. Open Security in your Cloudflare dashboard and record an explicit preference for Search, Agent and Training before the date. Doing nothing is still a decision.
Cloudflare is also extending Content Signals in robots.txt with a fourth field, use, which expresses how a crawler may reuse what it takes: immediate, reference, or full. It signals preference rather than enforcing anything, but crawlers that ignore it can lose Verified status. The generator adds a Content-Signal line for you when you block any training crawler.
Which AI crawlers can actually be blocked by IP
Not every bot in the picker can be enforced at the network layer, and the tools that pretend otherwise are quietly generating rules that do nothing. Here is the real state of every crawler on this page, read from each operator's own feed when this page loaded. The Source column links to the documentation each row is based on, so you can check the claim rather than take it.
| Crawler | Class | IP enforcement | Ranges | Feed | Source |
|---|---|---|---|---|---|
| GPTBotOpenAI | Training | Yes — published feed | 21 | 30 Oct 2025 (cached) | OpenAI bots |
| ClaudeBotAnthropic | Training | Yes — published feed | 26 | 18 Aug 2026 (cached) | Anthropic crawling |
| anthropic-aiAnthropic | Training | No — nothing published | — | — | Anthropic crawling |
| CCBotCommon Crawl | Training | Yes — published feed | 5 | 11 Aug 2026 (cached) | Common Crawl CCBot |
| BytespiderByteDance | Training | No — nothing published | — | — | None published |
| Google-ExtendedGoogle | Training | Impossible — no crawler | — | — | Google crawlers |
| Applebot-ExtendedApple | Training | Impossible — no crawler | — | — | Apple Applebot |
| Meta-ExternalAgentMeta | Training | Blunt — ASN only | — | — | Meta crawlers |
| AmazonbotAmazon | Training | Partly — web page only | — | — | Amazonbot |
| cohere-aiCohere | Training | No — nothing published | — | — | None published |
| DiffbotDiffbot | Training | No — nothing published | — | — | None published |
| OAI-SearchBotOpenAI | Search | Yes — published feed | 35 | 2 Jan 2026 (cached) | OpenAI bots |
| PerplexityBotPerplexity | Search | Yes — published feed | 8 | 7 Feb 2025 (cached) | Perplexity crawlers |
| Claude-SearchBotAnthropic | Search | Yes — published feed | 26 | 18 Aug 2026 (cached) | Anthropic crawling |
| Amzn-SearchBotAmazon | Search | Partly — web page only | — | — | Amazonbot |
| ChatGPT-UserOpenAI | Agent | Yes — published feed | 204 | 14 Aug 2026 (cached) | OpenAI bots |
| Perplexity-UserPerplexity | Agent | Yes — published feed | 4 | 17 Oct 2025 (cached) | Perplexity crawlers |
| Claude-UserAnthropic | Agent | Yes — published feed | 26 | 18 Aug 2026 (cached) | Anthropic crawling |
| Amzn-UserAmazon | Agent | Partly — web page only | — | — | Amazonbot |
| OAI-AdsBotOpenAI | Agent | No — nothing published | — | — | OpenAI bots |
Four rows in that table deserve a second look.
Google-Extended and Applebot-Extended cannot be blocked by IP at all
These are robots.txt tokens with no crawler behind them. They exist purely so you can opt out of Gemini and Apple Intelligence training without touching search. There is no address to add to a firewall. Any tool that hands you a CIDR rule for either one is making it up, and the only IP-level way to "enforce" it would be to block Googlebot or Applebot — which costs you Google Search and Siri results respectively.
Anthropic publishes one file for three crawlers
ClaudeBot, Claude-User and Claude-SearchBot all resolve to the same published range list, stamped August 2026. There is no way to block Anthropic training by IP while keeping Claude search and Claude user-fetches working. If you need that separation, robots.txt tokens are your only instrument. The generator flags this the moment your selection creates the conflict.
GPTBot and OAI-SearchBot share published prefixes
OpenAI does run separate crawlers with separate feeds, but the two feeds overlap on several prefixes. Block GPTBot at the IP layer and OAI-SearchBot loses those addresses too. The robots.txt tokens stay cleanly separate, so a training block via robots.txt keeps ChatGPT search intact in a way the firewall rule does not. Paste one of the shared addresses into the verification box above and you will see both crawlers returned for the same prefix.
Bytespider gives you nothing to work with
No documentation page, no published ranges, no reverse-DNS scheme, and a long record of fetching paths that robots.txt disallows. A Disallow line costs nothing to add, but do not expect it to be the thing that stops the traffic. If Bytespider is genuinely hurting your origin, rate limiting at the edge will do more than any list on this page.
How to deploy each output safely
Order matters here. Deploy the reversible thing first, watch what happens, then add enforcement.
1. robots.txt
Copy the robots.txt output and place it at the root of your site so it loads at yourdomain.com/robots.txt. If you already have a file there, merge the new groups in rather than overwriting — you will otherwise lose your sitemap line and any existing crawl rules. RFC 9309 tells crawlers not to use a cached copy for more than 24 hours, so most respect a change within a day. Amazon is the documented exception at up to 30 days.
2. Watch your logs for a week
Before adding firewall rules, search your access logs for the tokens you just blocked. If a crawler is still arriving in volume, the robots.txt layer did not work on it and the IP rule is worth adding. If it stopped, you may not need the second layer at all. This step catches the mistake where somebody blocks a crawler that was never visiting in the first place and then blames the tool.
3. Verify the addresses before you block them
Take a sample IP from those log lines and run it through the verification box at the top of this page. If it comes back verified, the crawler is real and a range rule will work. If it comes back with no published match and no reverse DNS, you are looking at something that only borrowed the name — and a rule built around that crawler's published ranges will never touch it. A full geolocation and network lookup on the address usually tells you what it really is.
4. nginx
The nginx output uses a geo block, which must sit in the http context — not inside server. Add it to nginx.conf or an included file, then reference the variable inside your server block as shown in the comment. Always run nginx -t before reloading; a geo block in the wrong context will refuse to start the service, and finding that out during a reload is an outage.
5. Apache .htaccess
Drop the output into the .htaccess file at your document root. The syntax uses mod_authz_core, which is Apache 2.4. On Apache 2.2 the directives are different and this block will produce a 500 error, so confirm your version with apachectl -v first. Shared hosting is nearly always 2.4 in 2026, but confirm rather than assume.
6. Cloudflare
Paste the expression into a custom rule under Security, set the action to Block, and deploy. Check your existing AI bot category settings first so you are not writing a rule that duplicates a managed one — and while you are in there, settle the legacy Block AI bots toggle described above. If the expression is long, the generator will tell you, and the fix is to move the ranges into a Cloudflare IP List and reference the list instead.
Automate the refresh. Every firewall output on this page is a snapshot of feeds that change. A small scheduled job that re-reads the vendor JSON and updates your IP list keeps the rule honest. If you paste once and forget, you are running a rule that gets less accurate every month — and it fails open, letting new crawler addresses straight through.
What this tool cannot do
Every generator on the internet will tell you what it does. Here is the other half, because deploying these rules while believing something untrue about them is worse than not deploying them.
It cannot stop a crawler that hides
Everything here depends on a crawler either declaring itself honestly or arriving from a published address. A scraper running a headless browser from a residential proxy, sending an ordinary Chrome user-agent, matches nothing on this page. That is not an AI crawler problem, it is a bot detection problem, and it needs behavioural analysis rather than a list.
It cannot remove you from a model already trained
Blocking a training crawler today affects future crawls. Content already collected is already collected, and Common Crawl archives from previous years remain public and remain in circulation. This is forward-looking protection only, and nobody can offer you the other kind.
The verification box has a hard ceiling
It answers whether an address sits in a published range and whether reverse DNS confirms it. It cannot verify a crawler whose operator publishes neither — Bytespider, Cohere and Diffbot will always come back unidentified, and that is a fact about them, not a failure of the check. It also cannot tell you what a verified crawler intends to do with the page.
It cannot keep your ranges fresh after you leave
The ranges are current while you are on this page. The moment you paste them into a config file they become a snapshot with an expiry date nobody prints on it. Perplexity's list has carried a February 2025 stamp for a long time now, so some of these were already old before you arrived.
It cannot make robots.txt binding
No tool can. RFC 9309 describes a voluntary protocol and says plainly that it is not a substitute for access control. A well-behaved crawler obeys it. A determined one reads it as a directory of what you would rather people not see.
It cannot tell you if blocking was the right call
The cost lines here describe what each block removes. Whether that trade is good for your business depends on where your traffic comes from, and this page has no view of that. If AI referrals already send you readers, blocking search crawlers is a revenue decision, not a security one.
One more, for completeness: none of these rules touch scrapers that reach you through a CDN or a cache. If your content is mirrored anywhere, blocking the origin does not block the mirror. You can check what your own edge reveals with our HTTP response header analyzer, and confirm which addresses your hostname actually resolves to with a DNS record lookup.
Frequently asked questions about blocking AI crawlers
How do I block AI crawlers in robots.txt?
Add a User-agent line naming the crawler and a Disallow line for the paths you want closed. To block OpenAI training, use User-agent: GPTBot followed by Disallow: /. Repeat the pair for each crawler and upload the file to your site root so it loads at yourdomain.com/robots.txt. Well-behaved crawlers read it before crawling, but it is a request rather than enforcement.
Does blocking GPTBot remove me from ChatGPT?
No. GPTBot collects training data. ChatGPT search results come from a separate crawler called OAI-SearchBot, and live user-triggered fetches come from ChatGPT-User. Blocking GPTBot in robots.txt keeps you eligible for ChatGPT search. An IP-based block is different: GPTBot and OAI-SearchBot publish several identical prefixes, so a firewall rule aimed at one will stop the other on those addresses.
Why is blocking by user-agent not enough?
A user-agent is a self-declaration. Any client can send the string GPTBot in a request header, and nothing about that string proves where the request came from. Matching the request IP against the operator’s published CIDR ranges is the only check a scraper cannot forge, which is why this generator outputs IP-range rules alongside the robots.txt lines. You can test any address against every published range using the verification box at the top of this page.
How do I opt out of AI training without losing search visibility?
Use the training-specific tokens and leave the search crawlers alone. Google-Extended opts you out of Gemini training while Googlebot keeps indexing you, and Applebot-Extended does the same for Apple Intelligence. Block GPTBot, ClaudeBot and CCBot, but allow OAI-SearchBot, PerplexityBot and Claude-SearchBot. Select Training only in the generator above and it builds exactly this configuration.
Can I block Google-Extended by IP address?
No. Google-Extended is a robots.txt token with no crawler behind it. It exists so you can opt out of Gemini and Vertex AI training without affecting Google Search, so there is no separate IP range to block. Blocking Googlebot instead would remove your site from Google Search entirely. Applebot-Extended works the same way for Apple Intelligence.
What changes on September 15, 2026 with Cloudflare?
Cloudflare sets new defaults that day. Training and Agent crawlers are blocked by default on pages that display ads, while Search crawlers stay allowed. The defaults reach new domains, new sites on existing accounts, and free-tier customers. Because the most restrictive applicable rule wins, blocking Training also blocks multi-purpose crawlers, and Cloudflare names Googlebot, Applebot and BingBot specifically.
I turned on Cloudflare’s old "Block AI bots" toggle years ago. Does that matter?
Yes, and this is the part that catches people out. Cloudflare’s announcement states the September 15 behaviour applies to customers who selected to block Training either through the new AI traffic controls or through the legacy Block AI bots service. If that old toggle is still on and you never think about it, Googlebot can be blocked on your ad-serving pages without anyone touching a dashboard. Open your zone Security settings and check it before the date.
Do AI crawlers actually obey robots.txt?
Most documented crawlers from OpenAI, Anthropic, Amazon and Common Crawl state that they honor it, and generally do. Bytespider is repeatedly reported fetching disallowed paths, and user-triggered agents may not apply robots.txt at all. RFC 9309 is explicit that the protocol is not a substitute for real access control, so treat firewall rules as the enforcement layer.
How often should I refresh AI crawler IP ranges?
Refresh at least monthly and automate it if you can. Vendors add and retire ranges as they move services between clouds, so a pasted list quietly goes stale: the rule starts letting new crawler addresses through while blocking addresses that now belong to somebody else. This page re-reads every published feed at most every six hours and prints the age of each one beside it.
Will blocking AI crawlers hurt my Google rankings?
Blocking the AI-specific tokens does not affect Google Search. Google-Extended controls Gemini training only, and Googlebot handles search indexing separately. The real risk is a rule that is too broad: a user-agent regex or an IP range that catches Googlebot, Bingbot or Applebot will cost you search visibility. Test any address your rule would block using the verification box above before you deploy it.
Related IP & network tools
Once you have a rule, these help you check what is actually hitting your server.
Browse the full set on the TrustMyIP tools directory.
Found a crawler you cannot identify?
If the verification box came back with no published match, the address belongs to something that only borrowed the name. Find out which network it really is before you decide what to do about it.
Last updated 26 August 2026 · IP ranges read from operator-published feeds · Nothing entered on this page is logged