Tick what you want crawlers to do and copy the file. A sitemap builder sits alongside it.
Allow Everything, Block Everything and Staging Site replace the rules below. Block Admin And Search and Block Common File Types add paths to the User-agent: * group.
robots.txt is a request, not a lock. These operators publish these names and say they honour it, but nothing enforces it: a crawler that ignores the file, or one that arrives under a name nobody publishes, reads the page anyway. Blocking at your server or CDN is the only thing that actually stops a bot.
GPTBot · OpenAI
Collects pages that may be used to train OpenAI's models.
OAI-SearchBot · OpenAI
Builds the results ChatGPT search shows. Block it and your pages stop being surfaced there.
ClaudeBot · Anthropic
Collects pages that may be used to train Anthropic's models.
Claude-SearchBot · Anthropic
Fetches pages so Claude can cite them when it searches the web.
PerplexityBot · Perplexity
Indexes pages so Perplexity can answer with a link back to the source.
Google-Extended · Google
Gemini training and grounding only. It has no effect on Google Search crawling, indexing or ranking.
CCBot · Common Crawl
The open web archive that many models, from many companies, are trained on.
Bytespider · ByteDance
ByteDance's crawler, used for its AI products.
Amazonbot · Amazon
Amazon's crawler, used to answer questions in Alexa and its AI products.
Applebot-Extended · Apple
Opts your pages out of Apple Intelligence training. Applebot itself keeps crawling for Siri and Spotlight.
meta-externalagent · Meta
Meta's crawler for training and for its AI products.
A Sitemap line is the one place in this file that takes a full web address rather than a path. It belongs to the file, not to any group, so it is written at the bottom.
No syntax problems found. Test a few paths below before you upload it.
/admin/settings for Googlebot. The longest rule that matches it in the * group is a Disallow, so the crawler is asked not to fetch it. The deciding line is Disallow: /admin/.
A crawler obeys one group only - the one whose name matches it best - and inside that group the longest matching rule wins, with Allow winning a tie. That is why Allow: / and Disallow: /admin/ can sit together.
Both files belong at the root of your site, so they answer at yoursite.com/robots.txt and yoursite.com/sitemap.xml - not in a folder, and not renamed. On OxyPages, a robots.txt and a sitemap.xml that you upload with your site are served from the root automatically. Everything on this page runs in your browser; nothing is uploaded until you press Publish.
How It Works
Pick Allow Everything, Block Everything, Staging Site, Block Admin And Search or Block Common File Types, then edit the groups underneath. Each group is a set of crawler names with its own Allow and Disallow paths.
Set each AI crawler to Allow or Block by its published name. The validator flags the mistakes that break a file - a missing User-agent, a path with no slash, a full URL where a path belongs, a stray wildcard, Disallow: / on a live site - and fixes each one in a click.
Type a path, pick a crawler and see whether it is allowed or blocked and which line decided. Switch to the sitemap tab, paste your page URLs, then copy or download robots.txt and sitemap.xml.
About This Tool
A robots.txt generator with the AI crawlers listed by name, a live validator, a real path tester, and a sitemap builder in the next tab.
robots.txt is a plain text file at the root of a site telling crawlers which parts they may read. sitemap.xml is the list of pages you would like them to know about. Neither is complicated, but both are unforgiving: a path without a leading slash matches nothing, a full web address where a path belongs matches nothing, and one stray Disallow: / takes an entire site out of search. This robots.txt generator writes both files from a form, checks them as you type, and lets you test a path against the rules before anything goes live.
Anyone who owns a site and has been told they need these files - a shop owner, a freelancer handing over a one-pager, someone who has just published a page built with an AI assistant and wants the admin area kept out of Google. You do not have to know the syntax: the presets cover the setups almost everyone needs, and every warning explains what breaks rather than quoting a spec at you.
Deciding whether AI companies may read your pages is now the main reason anyone opens this file, so the crawlers get a section of their own: GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, CCBot, Bytespider, Amazonbot, Applebot-Extended and meta-externalagent. Each has an Allow or Block toggle and a line saying what it actually does, because two are routinely misread: Google-Extended controls Gemini training and nothing about Google Search, and Applebot-Extended opts you out of Apple Intelligence training while Applebot keeps crawling for Siri. Be clear-eyed about the file itself, though. It is a request. These operators publish these names and say they honour it, but nothing enforces it, and a crawler arriving under a name nobody publishes reads your page regardless.
Everything runs in your browser: the file is assembled, validated and tested locally, and nothing is uploaded until you press Publish. The path tester implements the real matching rules - a crawler obeys one group, the longest matching pattern wins, Allow beats Disallow at equal length, and * and $ behave as Google and Bing document them - so it says why a page is blocked, not just that it is. The sitemap generator has one honest limit: it cannot crawl your site to discover pages, because crawling needs a server. Paste the list instead. Worth knowing too: blocking a URL here does not hide it from search results - that needs a noindex meta tag on the page itself - and if the HTML those crawlers reach is broken, the HTML Validator is the faster thing to run first.
Both files belong at the root of a site, which is the part people get stuck on: not in a folder, not renamed, and reachable at yoursite.com/robots.txt. Publish A Share Page puts both finished files on a live OxyPages link with HTTPS in seconds and without an account - the easiest way to hand them to whoever looks after the server. The link lasts 30 minutes unless you claim it, and claiming is free. If you host the site itself on OxyPages, a robots.txt and a sitemap.xml uploaded with it are served from the root automatically. See how it works.
Questions
More Free Tools
Every tool here runs in your browser and ends the same way: one click and your HTML is on a live link.
Generate title, description, Open Graph and Twitter card tags for your page and see how it will look on Google, Facebook, X and WhatsApp before you publish.
Check HTML for unclosed tags, duplicate IDs, missing alt text and other mistakes, with line numbers. Runs in your browser. Fix it, then publish it live.
Break a long URL into its parts, read and edit the query string, decode percent-encoding and international domains, then copy the rebuilt link. Free and in-browser.
Try It Now
Still Here? Drop It In.
The whole pitch fits in one sentence: your HTML, on a link, in seconds.
Drag and drop your HTML file(s), folders, or ZIP file
or ·
No account needed. Your unclaimed website stays live for 30 minutes on a free subdomain. Claim it to your account to keep it permanently.
