Which AI Bots Are Crawling Your WordPress Site (and How to Control Them)

Which AI bots are crawling websites in 2026?

Dozens of AI companies now run crawlers, but a handful account for most of the traffic. The names you will see most often in your logs are:

• GPTBot and OAI-SearchBot from OpenAI.
• ClaudeBot, anthropic-ai and Claude-SearchBot from Anthropic.
• PerplexityBot and Perplexity-User from Perplexity.
• Google-Extended (Google's AI training control token) and Googlebot for classic search.
• CCBot (Common Crawl), Bytespider (ByteDance), Amazonbot, Applebot-Extended and Meta-ExternalAgent.

New crawlers appear every few months, so the exact list shifts. What does not change is that they fall into three groups, and knowing the group tells you what to do about each bot.

The three types of AI bot: Training, AI-Search and Agent

Every AI crawler is doing one of three jobs. This is the single most useful way to think about them, because it maps directly to whether the bot gives anything back.

• Training bots take your content to train future models. They rarely send traffic back. GPTBot, ClaudeBot, CCBot and Bytespider live here. These are the ones owners most often consider blocking.
• AI-Search bots power live AI answers and can cite you and send referral traffic. OAI-SearchBot, PerplexityBot and Claude-SearchBot are in this group. Usually worth allowing, because a citation is a visit.
• Agent bots fetch a page because a real person just asked an AI to visit it. ChatGPT-User and Perplexity-User are agents. This is the closest thing to a human visitor.

Once you sort a bot into one of these buckets, the decision is simple. Training takes and gives little, AI-Search can give back, and Agent is a real person right now.

How do I see which AI bots visit my WordPress site?

There are two ways to see AI crawler traffic on WordPress. The manual way is to read your server access logs and filter for the user-agent strings above. The easier way is a crawler log that records each AI bot hit for you, with the date, the page and the bot's category.

A good crawler log answers three questions at a glance: who visited, what they read, and whether they hit any dead pages. It runs entirely on your own site, so there is nothing to send anywhere. You want to know, for example, that GPTBot crawled forty pages last week while PerplexityBot only read three, because that tells you who is taking your content and who might actually cite it.

Should you block GPTBot and the other training bots?

There is no single right answer, and it depends on your goal. Blocking training bots stops your content from being used to train models, but it does not help you get cited. Allowing them does not directly send traffic either. It is a values-and-strategy call, not a technical one.

• Block training bots if you do not want your work used to train commercial models and you see no benefit in return.
• Allow AI-Search bots in almost all cases, because they are how your content gets cited in ChatGPT Search, Perplexity and similar answers.
• Allow Agent bots, since blocking them means blocking a real person who asked their AI to open your page.

A common middle path is to block the pure training crawlers, keep the AI-Search and Agent bots allowed, and watch your logs to see if anything changes.

How to allow or block AI bots with robots.txt

You control AI crawlers the same way you control any crawler: with robots.txt. To block a bot, add a User-agent block with a Disallow rule. For example, to block OpenAI's training crawler and Common Crawl while leaving everything else allowed:

• User-agent: GPTBot then Disallow: /
• User-agent: CCBot then Disallow: /

Leave a bot out of robots.txt entirely, or give it Disallow: with no path, and it stays allowed. Keep your Sitemap: line in the file so search and AI-Search engines can still discover your pages. Editing robots.txt by hand works, but it is easy to typo a user-agent name and silently block the wrong bot, so check your file after every change.

Does robots.txt actually stop AI bots?

robots.txt is a request, not a wall. Reputable operators such as OpenAI, Anthropic, Google and Perplexity honor it. Some crawlers ignore it entirely, and agentic browsers that visit as a person may not identify themselves at all.

So use robots.txt to steer the well-behaved majority, and do not assume it hard-blocks anyone. For genuine enforcement, block at the network edge with a firewall or a CDN like Cloudflare, which can challenge or drop specific bots regardless of what robots.txt says. Be honest with yourself about the difference: robots.txt sets policy, a firewall enforces it.

What about the dead pages AI bots hit?

AI crawlers frequently request URLs that no longer exist, often old links they learned months ago. Every one of those is a wasted crawl and a missed chance to be read. Two fixes cover almost all cases:

• Recreate or republish the page if the content still matters.
• Add a 301 redirect from the dead URL to the closest live page, so the bot lands somewhere useful.

A crawler log that flags the 404s AI bots hit turns this into a short, concrete to-do list instead of a guessing game.

How wptaskify shows and controls your AI crawlers

wptaskify includes a free AI crawler log built for exactly this. It records every AI bot hit on your WordPress site, sorts each one into Training, AI-Search or Agent, shows the dead URLs bots requested, and gives you one-click AI Access Control that writes the correct robots.txt rules for you. You decide who to allow or block, and the plugin keeps your own robots.txt rules untouched. To go further and have your own AI fix the dead links or restructure pages, connect your site to Claude or ChatGPT.