On a PBN, allow search engines and AI crawlers (Googlebot, Bingbot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot and friends) and block backlink crawlers like AhrefsBot, SemrushBot, MJ12bot, DotBot and BLEXBot. Blocking Google-Extended only opts you out of Gemini training; it does not affect Google Search or AI Overviews. Then check your CDN isn’t blocking AI bots before robots.txt is ever read.
Last updated: October 2026. Prices exclude tax and may change. We may earn a commission if you sign up through our links.
robots.txt is a small file with outsized consequences on a private blog network. Get it wrong one way and AI assistants never see your content. Get it wrong the other way and every competitor can map your network in an afternoon with a backlink tool. This guide gives you a working file, explains each line and covers the layer most people forget: bot blocking at the CDN.
Which bots should a PBN allow?
Anything that sends you rankings, citations or traffic. In 2026 that means two groups.
Search engine crawlers
- Googlebot: Google Search, including AI Overviews.
- Bingbot: Bing Search. Bing’s index also feeds ChatGPT search and Copilot, so blocking it costs you AI visibility too.
AI crawlers and fetchers
| User agent | Operator | What it does |
|---|---|---|
| GPTBot | OpenAI | Collects content for model training |
| OAI-SearchBot | OpenAI | Indexes pages for ChatGPT search |
| ChatGPT-User | OpenAI | Fetches a page when a user’s request needs it |
| ClaudeBot | Anthropic | Collects content for training |
| Claude-SearchBot | Anthropic | Indexes pages for Claude’s search |
| Claude-User | Anthropic | Fetches a page on a user’s behalf |
| PerplexityBot | Perplexity | Indexes pages for Perplexity answers |
| Perplexity-User | Perplexity | Fetches a page on a user’s behalf |
The search and user-fetch bots are the ones that get you cited in live answers. The training bots (GPTBot, ClaudeBot) are a judgment call. Allowing them means your content can shape what future models “know”. For most PBN owners who want topical authority to spread, allowing them is the sensible default.
Which bots should you block?
Backlink crawlers. They build the link databases that competitors, and sometimes manual reviewers, use to spot networks. The common ones:
- AhrefsBot (Ahrefs)
- SemrushBot (Semrush)
- MJ12bot (Majestic)
- DotBot (Moz)
- BLEXBot
Blocking them won’t hide your links from Google. It hides them from the tools people use to find PBNs.
What does a good PBN robots.txt look like?
Here is a starting file. Adjust the sitemap URL per site.
# Search engines
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /
# AI search, user fetchers and training crawlers
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: GPTBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
# Backlink crawlers
User-agent: AhrefsBot
Disallow: /
User-agent: SemrushBot
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: DotBot
Disallow: /
User-agent: BLEXBot
Disallow: /
# Everyone else
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/sitemap.xml
A few notes on this file:
- Named groups override the
*group, so a bot with its own block ignores the generic rules. - We left Google-Extended out on purpose. See below.
- Don’t copy the exact same file, with the same comments and ordering, to every site in your network. Identical files are a minor footprint. Vary the order, drop the comments, or use different generic rules per site.
Is robots.txt enough to block backlink crawlers?
Not quite. robots.txt is a request, not a lock. The major backlink tools say they respect it, but anyone can read your robots.txt and see that you’re blocking Ahrefs and Majestic, which itself hints that the site has something to hide. A list of five SEO-tool blocks on a hobby blog about gardening looks odd.
Server-level blocking is quieter. The request is refused by user agent, and robots.txt can stay plain. Options include:
- Host-level blocking. LaunchCDN blocks backlink checkers by default while allowing Googlebot, Bingbot and AI crawlers.
- A plugin. Bulk Buy Hosting leaves backlink-checker blocking off by default and offers a free Link Silencer plugin you can install per site. AI crawlers are allowed.
- .htaccess or server rules on any Apache host, matching user agents and returning 403.
Pick one approach per site and vary it across the network if you can.
What does Google-Extended actually control?
This is the most misunderstood token in robots.txt. Google-Extended is not a crawler. It’s a product token that tells Google whether content Googlebot already crawled may be used for Gemini model training. Blocking it:
- Does not remove you from Google Search.
- Does not remove you from AI Overviews.
- Does opt you out of Gemini training use.
So “block Google-Extended to protect rankings” is a myth. If you want your content to influence Gemini, leave it alone. If you’d rather opt out of training, block it, knowing Search is unaffected.
Can your CDN block AI crawlers without you knowing?
Yes, and this catches a lot of people. CDNs and firewalls can block bots before the request reaches your server, so robots.txt never gets a say. Cloudflare in particular offers an AI-crawler blocking setting that can be switched on by default for domains added to it. Your robots.txt says “Allow”, the AI bot gets a block page, and you never see it in your logs.
How to check:
- In your CDN dashboard, look for bot management, AI crawler or “AI scrapers” settings on each zone.
- Test with a user agent switcher:
curl -A "GPTBot" -I https://example.com/should return 200, not 403. - Look at your access logs for AI user agents. Total silence over several weeks is a warning sign.
If you host on LaunchCDN, this is handled for you: it switches off Cloudflare’s AI-crawler blocking on the sites it deploys. If you put your own Cloudflare account in front of any host, the setting is yours to check.
Should every PBN site use the same robots.txt?
The rules can be the same; the file shouldn’t be byte-identical. Footprints come from repetition: same theme, same plugins, same robots.txt with the same comment lines. Small variations cost nothing. For the broader picture, read our guide to reducing your digital footprint.
Which host makes this easiest?
Both hosts on our shortlist allow AI crawlers. The difference is defaults:
- LaunchCDN, best for large networks & CDN hosting, blocks backlink checkers out of the box and keeps Cloudflare’s AI blocking off.
- Bulk Buy Hosting, best for real shared-hosting IPs with cPanel, leaves you in control with the free Link Silencer plugin.
Using both is best for maximum IP diversity, and having different blocking methods across the two halves of a network is a small bonus. See the full comparison, or read how to get recommended by ChatGPT, Claude and Perplexity once the bots can reach you.
Frequently Asked Questions
Does blocking Google-Extended hurt my Google rankings?
No. Google-Extended only controls whether your content is used for Gemini training. It does not affect Google Search or AI Overviews.
Should I block GPTBot and ClaudeBot on a PBN?
They are training crawlers, so it is a judgment call. Most PBN owners who want their topical content to spread allow them, along with the search and user-fetch bots.
Which backlink crawlers should a PBN block?
The common ones are AhrefsBot, SemrushBot, MJ12bot (Majestic), DotBot (Moz) and BLEXBot.
Why are AI bots blocked even though my robots.txt allows them?
A CDN or firewall may be blocking them before robots.txt is read. Cloudflare has an AI-crawler blocking setting that can be on by default, so check each zone and test with a GPTBot user agent.
Is robots.txt the best way to block backlink checkers?
It works for compliant bots but is publicly readable, which can itself look suspicious. Server-level or plugin blocking by user agent is quieter.