How to Create Your robots.txt File
- Start from a preset: allow everything, block only AI training bots, block all AI bots, WordPress defaults or block everything for a staging site.
- Add Disallow paths for areas that don't belong in search results, such as admin pages, carts, internal search results and filter parameters.
- Set individual bots to Allow or Block where you want different rules from the default.
- Add your sitemap URL and test a few important URLs with the tester.
- Download robots.txt and upload it to the root of your domain so it's available at https://yourdomain.com/robots.txt.
What Is robots.txt and How Do Web Crawlers Process It?
robots.txt is a plain-text file at the root of your website that tells crawlers which URLs they may request. It's defined by the Robots Exclusion Protocol, standardised as RFC 9309. A file contains one or more groups, each starting with one or more User-agent lines followed by Disallow and Allow rules. A crawler uses the group that names it most specifically, falling back to the * group. Within that group, the longest matching rule wins, and if an Allow and a Disallow rule are equally long, Allow wins. * matches any sequence of characters, and $ marks the end of a URL. An empty Disallow: means everything is allowed. The tester applies exactly these rules.
robots.txt controls crawling, not indexing. A blocked page can still appear in Google if other sites link to it. To keep a page out of search results, allow crawling and add a noindex meta tag, or protect the page with a password.
Blocking AI Scrapers vs Search Engine Crawlers
Since 2023, AI companies have introduced separate crawlers, and many now split them by purpose:
| Purpose | Examples | If you block it |
|---|---|---|
| AI training | GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider | Your future content isn't used to train those models. No effect on Google or Bing rankings. |
| AI search | OAI-SearchBot, Claude-SearchBot, PerplexityBot | Your pages may stop appearing, or being cited, in AI search answers. |
| User-triggered | ChatGPT-User, Claude-User, Perplexity-User | AI assistants can't fetch your page when a user asks about it. |
The "Block AI training only" preset is a popular middle ground: it opts out of model training while keeping your site visible and linked in AI-powered search. Note that Google-Extended doesn't affect Google Search or AI Overviews, which rely on Googlebot, so blocking it won't remove you from Google. Each bot needs its own rule: blocking ClaudeBot doesn't block Claude-SearchBot or Claude-User, and blocking GPTBot doesn't block OAI-SearchBot.
Common robots.txt Mistakes
- Blocking the whole site by accident with
Disallow: /left over from a staging site. - Blocking CSS and JavaScript, which stops Google rendering your pages.
- Using robots.txt to hide private content. The file is public, and it points people straight to what you're trying to hide.
- Forgetting that paths are case-sensitive: /Admin/ and /admin/ are different.
- Relying on Crawl-delay for Google. Google ignores it; manage crawl rate in Search Console instead.
Real-World Example
A recipe blog wanted to stay in Google and appear in ChatGPT and Perplexity answers, but not have its recipes used to train AI models. Using the "Block AI training only" preset plus /search? and /*?sort= to stop crawlers wasting time on duplicate filter pages, it produced a clean robots.txt with one group blocking the training bots, one group explicitly allowing AI search and user bots, and default rules plus the sitemap for everyone else. In the tester, GPTBot is blocked on /recipes/lasagne while OAI-SearchBot and Googlebot are allowed.
Frequently Asked Questions
Where do I put robots.txt?
In the root folder of your domain, so it loads at https://yourdomain.com/robots.txt. Each subdomain needs its own file.
Does blocking GPTBot or ClaudeBot affect my Google rankings?
No. They're separate from search engine crawlers, so blocking them has no effect on Google or Bing.
Do all bots obey robots.txt?
Major search engines and the AI companies listed say they respect it. Malicious scrapers may ignore it; use a firewall or bot management to block those.
How long until changes take effect?
Google usually re-reads robots.txt within 24 hours. You can request a refresh in Search Console's robots.txt report.
What's the difference between Disallow and noindex?
Disallow stops crawling; noindex stops indexing. To remove a page from search results, use noindex and don't block the page in robots.txt, or crawlers won't see the noindex tag.