AI bots crawl WordPress sites for three reasons: to train models (GPTBot, ClaudeBot, CCBot, Bytespider), to power live AI answers that can cite you (OAI-SearchBot, PerplexityBot), and to fetch a page a real user asked an AI to visit (ChatGPT-User, Perplexity-User). You can see them in your server logs or a crawler log plugin, and allow or block each one in robots.txt.
Which AI bots are crawling websites in 2026?
Dozens of AI companies now run crawlers, but a handful account for most of the traffic. The names you will see most often in your logs are:
- GPTBot and OAI-SearchBot from OpenAI.
- ClaudeBot, anthropic-ai and Claude-SearchBot from Anthropic.
- PerplexityBot and Perplexity-User from Perplexity.
- Google-Extended (Google's AI training control token) and Googlebot for classic search.
- CCBot (Common Crawl), Bytespider (ByteDance), Amazonbot, Applebot-Extended and Meta-ExternalAgent.
New crawlers appear every few months, so the exact list shifts. What does not change is that they fall into three groups, and knowing the group tells you what to do about each bot.
The three types of AI bot: Training, AI-Search and Agent
Every AI crawler is doing one of three jobs. This is the single most useful way to think about them, because it maps directly to whether the bot gives anything back.
- Training bots take your content to train future models. They rarely send traffic back. GPTBot, ClaudeBot, CCBot and Bytespider live here. These are the ones owners most often consider blocking.
- AI-Search bots power live AI answers and can cite you and send referral traffic. OAI-SearchBot, PerplexityBot and Claude-SearchBot are in this group. Usually worth allowing, because a citation is a visit.
- Agent bots fetch a page because a real person just asked an AI to visit it. ChatGPT-User and Perplexity-User are agents. This is the closest thing to a human visitor.
Once you sort a bot into one of these buckets, the decision is simple. Training takes and gives little, AI-Search can give back, and Agent is a real person right now.
How do I see which AI bots visit my WordPress site?
There are two ways to see AI crawler traffic on WordPress. The manual way is to read your server access logs and filter for the user-agent strings above. The easier way is a crawler log that records each AI bot hit for you, with the date, the page and the bot's category.
A good crawler log answers three questions at a glance: who visited, what they read, and whether they hit any dead pages. It runs entirely on your own site, so there is nothing to send anywhere. You want to know, for example, that GPTBot crawled forty pages last week while PerplexityBot only read three, because that tells you who is taking your content and who might actually cite it.
Should you block GPTBot and the other training bots?
There is no single right answer, and it depends on your goal. Blocking training bots stops your content from being used to train models, but it does not help you get cited. Allowing them does not directly send traffic either. It is a values-and-strategy call, not a technical one.
- Block training bots if you do not want your work used to train commercial models and you see no benefit in return.
- Allow AI-Search bots in almost all cases, because they are how your content gets cited in ChatGPT Search, Perplexity and similar answers.
- Allow Agent bots, since blocking them means blocking a real person who asked their AI to open your page.
A common middle path is to block the pure training crawlers, keep the AI-Search and Agent bots allowed, and watch your logs to see if anything changes.
How to allow or block AI bots with robots.txt
You control AI crawlers the same way you control any crawler: with robots.txt. To block a bot, add a User-agent block with a Disallow rule. For example, to block OpenAI's training crawler and Common Crawl while leaving everything else allowed:
User-agent: GPTBotthenDisallow: /User-agent: CCBotthenDisallow: /
Leave a bot out of robots.txt entirely, or give it Disallow: with no path, and it stays allowed. Keep your Sitemap: line in the file so search and AI-Search engines can still discover your pages. Editing robots.txt by hand works, but it is easy to typo a user-agent name and silently block the wrong bot, so check your file after every change.
Does robots.txt actually stop AI bots?
robots.txt is a request, not a wall. Reputable operators such as OpenAI, Anthropic, Google and Perplexity honor it. Some crawlers ignore it entirely, and agentic browsers that visit as a person may not identify themselves at all.
So use robots.txt to steer the well-behaved majority, and do not assume it hard-blocks anyone. For genuine enforcement, block at the network edge with a firewall or a CDN like Cloudflare, which can challenge or drop specific bots regardless of what robots.txt says. Be honest with yourself about the difference: robots.txt sets policy, a firewall enforces it.
What about the dead pages AI bots hit?
AI crawlers frequently request URLs that no longer exist, often old links they learned months ago. Every one of those is a wasted crawl and a missed chance to be read. Two fixes cover almost all cases:
- Recreate or republish the page if the content still matters.
- Add a 301 redirect from the dead URL to the closest live page, so the bot lands somewhere useful.
A crawler log that flags the 404s AI bots hit turns this into a short, concrete to-do list instead of a guessing game.
How wptaskify shows and controls your AI crawlers
wptaskify includes a free AI crawler log built for exactly this. It records every AI bot hit on your WordPress site, sorts each one into Training, AI-Search or Agent, shows the dead URLs bots requested, and gives you one-click AI Access Control that writes the correct robots.txt rules for you. You decide who to allow or block, and the plugin keeps your own robots.txt rules untouched. To go further and have your own AI fix the dead links or restructure pages, connect your site to Claude or ChatGPT.
Frequently asked questions
How do I know if AI is scraping my website?
Check your server access logs or an AI crawler log plugin for user-agents like GPTBot, ClaudeBot, PerplexityBot and CCBot. Each hit is an AI bot reading a page on your site.
Is blocking GPTBot a good idea?
It depends on your goal. Blocking GPTBot stops OpenAI training on your content but does not affect being cited in ChatGPT Search, which uses OAI-SearchBot. Many owners block training bots and keep AI-Search bots allowed.
Will blocking AI crawlers hurt my SEO?
Blocking AI training bots does not affect Google search rankings, which use Googlebot. But blocking AI-Search bots like OAI-SearchBot or PerplexityBot means those engines cannot cite you, so weigh that before blocking them.
Can I block AI bots without editing robots.txt by hand?
Yes. A plugin with AI Access Control writes the correct robots.txt rules for the bots you choose to block and leaves your own rules alone, so you avoid typos and blocking the wrong crawler.
Related guides
See which AI bots crawl your site and control them, free
Bring your own Claude or ChatGPT and let AI write, optimize, sell and publish for you.