For most small businesses, the best answer is to block AI crawlers selectively, not all at once. Allow the bots that put your business in AI search answers, decide separately whether AI companies may train on your content, and block or slow down any scraper that ignores the rules or strains your server. OpenAI, Anthropic and Google now let you control these jobs separately.
Key takeaways
- AI search bots and AI training bots are different, and robots.txt can treat them differently.
- Blocking AI search bots can remove you from answers in ChatGPT and Claude, where some customers now ask for recommendations.
- Robots.txt is a request, not a lock. Bad bots need a firewall or server rules.
- Blocking Google-Extended doesn’t affect your Google Search rankings.
What are AI crawlers, and what do they want?
AI crawlers are automated programs that read web pages for AI companies. They do three different jobs:
- Training: collecting content to train future AI models. Examples are OpenAI’s GPTBot and Anthropic’s ClaudeBot.
- Search: indexing pages so an AI assistant can find and cite them in answers. Examples are OAI-SearchBot and Claude-SearchBot.
- User requests: fetching a page because a person asked the assistant about it, such as ChatGPT-User and Claude-User.
OpenAI says each of its settings “is independent of the others”, so a site can allow OAI-SearchBot while blocking GPTBot. Sites that block OAI-SearchBot “will not be shown in ChatGPT search answers” (OpenAI). Anthropic’s three Claude bots work the same way, and it says they honour robots.txt (Anthropic).
Should you block AI crawlers?
Weigh up three things: visibility, your content and your server.
Visibility
Some customers now ask AI assistants which local business to use. If the search bots can’t read your site, you can’t appear in those answers. For a tradie, clinic or retailer, that’s usually a reason to allow them. Our guide to AI search optimisation explains how to show up.
Your content
If your value is original writing, research or courses, you may not want it training models that answer for you. Cloudflare argues AI has made traffic far harder to earn. It says traffic from OpenAI is about 750 times harder to earn than it was from the Google of old (Cloudflare). Blocking training bots is a fair choice in that case.
Server load
Some crawlers request thousands of pages quickly. On a small hosting plan, that can slow the site for real visitors. This is less about AI and more about bad manners, and it’s a hosting problem with a hosting fix.
How do I block AI crawlers with robots.txt?
Robots.txt is a plain text file at the root of your site. To block training bots while staying visible in AI search, add rules like these:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /Leave OAI-SearchBot, Claude-SearchBot and Googlebot alone. OpenAI says changes for search can take about 24 hours to apply.
Google-Extended controls whether Google can use your content to train Gemini. Google states it “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal” (Google Search Central). It doesn’t control AI Overviews, which follow your normal Googlebot and snippet settings (Google).
Tip: On WordPress, an SEO plugin or your host can usually edit robots.txt for you. Check the result by visiting yoursite.com.au/robots.txt in a browser.
Why doesn’t robots.txt stop every bot?
Because it’s voluntary. Google is blunt about it: robots.txt “cannot enforce crawler behavior to your site; it’s up to the crawler to obey them” (Google Search Central). The big AI companies say they follow it. Unknown scrapers often don’t, and some pretend to be a normal browser.
Those bots need blocking at the firewall or server, by user agent, behaviour or request rate. A WordPress firewall helps here, and ongoing website security monitoring spots unusual traffic before it slows the site down.
And if content is truly private, such as client files, don’t rely on robots.txt at all. Put it behind a login. Our guide to website hosting for accountants covers confidential material in more detail.
What can a host do that robots.txt can’t?
A good host sees every request before WordPress does. That means it can:
- show you which bots are visiting and how often
- block a bad bot by its user agent or address before it reaches your site
- limit how fast any single visitor can request pages
- keep your plan’s resources free for real customers
This is part of what we mean by secure web hosting. If you host with us, tell us which bots you want allowed or blocked and we’ll help you set it up. For a wider checklist, see what makes web hosting secure.
What we’d suggest for most small businesses
- Allow AI search bots. Being recommended by an assistant is free visibility.
- Decide on training bots. Block them if your content is your product. Otherwise, it’s a matter of preference.
- Block anything abusive at the server. Don’t let one scraper slow the site for everyone else.
- Review every few months. New bots appear, and companies add new ones for new jobs.
Whatever you decide, the decision should sit with you, not with a default you never chose. Compare our hosting plans if your current host can’t tell you who’s crawling your site.
Frequently asked questions
Will blocking GPTBot remove my site from ChatGPT?
Not from ChatGPT search. GPTBot is OpenAI’s training crawler. ChatGPT search uses a separate bot, OAI-SearchBot, which you can keep allowed. OpenAI says the two settings are independent, so you can opt out of training and stay visible in search answers.
Does blocking AI crawlers hurt my Google rankings?
Blocking Google-Extended, GPTBot or ClaudeBot doesn’t affect Google Search. Google says Google-Extended isn’t a ranking signal. Blocking Googlebot itself would hurt your rankings, so never block it to stop AI use.
Can I stop my content appearing in Google’s AI Overviews?
Only with the normal search controls, such as nosnippet or noindex. Those also limit how your pages appear in regular results. For most businesses the trade-off isn’t worth it, because AI Overviews can still send visitors your way.
How do I know which AI bots are visiting my site?
Your server’s access logs record each visitor’s user agent, such as GPTBot or ClaudeBot. Your host can pull a summary for you. Some security plugins and services such as Cloudflare also show bot traffic in their dashboards.
Do AI crawlers slow down my website?
Well-behaved ones usually don’t. Aggressive scrapers can, especially on small plans with limited CPU and memory. If your site slows at odd hours with no extra customers, check the logs for bot traffic and block or rate-limit the worst offenders.
Sources
- Overview of OpenAI crawlers, OpenAI
- Does Anthropic crawl data from the web, and how can site owners block the crawler?, Anthropic
- Content Independence Day: no AI crawl without compensation, Cloudflare, 1 July 2025
- Google’s common crawlers, Google Search Central
- Introduction to robots.txt, Google Search Central
- AI features and your website, Google Search Central