FIRST CH TOOLS LOCAL PROCESSING — NOTHING LEAVES YOUR DEVICE 日本語

FIRST CH TOOLS / 36 ROBOTS.TXT GENERATOR

robots.txt Generator

Fill in a form and get a robots.txt that tells search engines and AI crawlers where they may go. Presets cover the common request — turn away training crawlers such as GPTBot and ClaudeBot while letting AI search engines in. Upload the file to your site root (https://example.com/robots.txt).

1 — Basic rules

2 — AI crawlers

A crawler obeys exactly one matching group. As soon as an AI crawler has its own group, the User-agent: * rules stop reaching it — so it is best to leave the checkbox above on.

3 — Sitemaps

4 — Output

robots.txt 

    

      
      
    

  


  

    
0
Lines

    
0
Allow / Disallow

    
0
AI crawlers blocked

    
0
Bytes

  


  

    
What we noticed

    
  


  
Nothing you type leaves the browser — the file is assembled and checked on this page. You can also call the tool from a URL: /en/robots-txt/?site=https%3A%2F%2Fexample.com&ai=training or /en/robots-txt/?policy=custom&disallow=/admin/,/cart/&sitemap=https%3A%2F%2Fexample.com%2Fsitemap.xml


  

    
How to Use

    
    
      
  1. Pick a policyAllow the whole site, or list the paths to keep out of the index. Admin screens, carts and search results can be added with the quick-add buttons.
    2. 
      
  2. Decide on AI crawlersThe preset switches between block training only, allow everything and block everything. You can also set each crawler by hand.
    3. 
      
  3. Upload to the rootDownload the file and put it at the top level of your site (https://example.com/robots.txt). A copy in a subdirectory is never read.
    4. 
    

  


  

    
About This Tool

    
robots.txt is the note you leave for crawlers at the door. Search engines and AI crawlers read this text file at the site root before anything else. User-agent names the crawler, Disallow lists what it must not fetch and Allow carves out exceptions. It was a de facto standard for nearly thirty years and became a real specification, RFC 9309, in 2022.

    
"Don't train on my site, but do cite it" is a setting here. AI crawlers come with three different jobs: collecting training data (GPTBot, ClaudeBot, Google-Extended), building an AI search index (OAI-SearchBot, PerplexityBot, Claude-SearchBot) and fetching on demand when a user pastes a link (ChatGPT-User). This tool groups them by job, so the most common request — refuse training, keep the AI search citations — is one preset away. Pair it with the llms.txt generator to state both who is turned away and who is welcome.

    
It closes the gap a dedicated group opens. A crawler follows one matching group and ignores the rest. So Disallow: /admin/ under User-agent: * stops applying to GPTBot the moment a User-agent: GPTBot group exists. By default this tool copies the shared disallow paths into every AI group it writes, so nothing leaks through.

    
Mistakes are flagged as you type. Missing leading slashes, full URLs pasted where a path belongs, spaces inside a path, non-ASCII paths that need percent-encoding, the same User-agent in two groups, a Crawl-delay that Googlebot will ignore, a missing sitemap — all of it appears under "What we noticed". Leading slashes and pasted URLs are fixed for you.

    
Once the file is live, check it with the robots.txt report in Search Console. Related tools: llms.txt, JSON-LD generator and the OGP meta tag wizard.

  


  

    
Where robots.txt trips people up

    

      

        
Disallow does not remove a page from search

        
Disallow: /secret/     ← only stops the fetch
<meta name="robots" content="noindex">   ← this is what removes it

        
Disallow keeps a crawler from fetching the page. If another site links to it, the URL can still be listed without its contents. Worse, blocking a page that carries noindex means the crawler never reads the noindex, so the page stays. To drop a page from search, let it be crawled and mark it noindex.

      

      

        
It is not a place to hide URLs

        
Disallow: /admin-secret-2026/   ← published the address to everyone

        
robots.txt is a public file. Naming a directory you want hidden is the same as announcing it. Anything that must not be reached needs a password, authentication or an IP restriction. robots.txt is a request to well-behaved crawlers and nothing more; hostile ones ignore it.

      

      

        
A crawler reads one group and only one

        
User-agent: *
Disallow: /admin/

User-agent: GPTBot
Disallow:          ← GPTBot never sees /admin/ above, so it may crawl it

        
A crawler looks for the group naming it and, once found, follows that group alone. The * group is the fallback for crawlers with no group of their own. Whenever you write an exception for one crawler, repeat the shared rules inside it (this tool does that by default).

      

      

        
Blocking CSS and JS hurts the page

        
Disallow: /assets/    ← the page is judged as a broken layout

        
Googlebot renders the page before judging it. Blocking stylesheets, JavaScript and images gets the page evaluated in its broken state, which also affects mobile-friendliness. Old advice to block /wp-includes/ or /assets/ wholesale no longer holds.

      

      

        
The location and the file name are fixed

        
https://example.com/robots.txt        ← the only file that is read
https://example.com/sub/robots.txt    ← ignored
https://sub.example.com/robots.txt    ← a separate file for the subdomain

        
There is one robots.txt per host, at the root. A copy in a subdirectory is ignored, and subdomains (blog.example.com) and schemes (http vs https) each get their own file. The name is lowercase robots.txt; mixed case is not read.

      

      

        
Wildcards and $ are an extension, not a guarantee

        
Disallow: /*?utm_source=    ← URLs carrying a query parameter
Disallow: /*.pdf$           ← URLs ending in .pdf

        
* (any run of characters) and $ (end of URL) are in RFC 9309 and understood by Google, Bing and OpenAI, but not by every crawler. A crawler without support reads them as literal characters, and the exclusion silently does nothing.

      

      

        
Googlebot ignores Crawl-delay

        
Crawl-delay: 10   ← honoured by Bing and Yandex, ignored by Google

        
People add Crawl-delay to spare the server, but Googlebot does not read it (Google crawl rate is a Search Console setting). Bing and Yandex do follow it. Keep the value small — a large delay slows indexing to a crawl.

      

      

        
Anything unnamed is allowed

        
User-agent: GPTBot
Disallow: /
(every crawler not named here is welcome)

        
robots.txt permits whatever it does not mention. Crawlers you did not name — including ones that launch next month — are all allowed. Blocking "AI in general" means a rule under User-agent: *, and that stops search engines too. How far to go is a business decision, not a technical one.

      

    

  


  

    
From AI Agents

    
This generation logic is also available as the robotstxt_generate tool in our MCP (Model Context Protocol) server @first-ch/tools-mcp, callable directly by AI agents with no browser needed. See Using These Tools from AI Agents for setup details.

    
Install

    
claude mcp add firstch-tools -- npx -y @first-ch/tools-mcp

    
Example call

    
# write a robots.txt that turns away training crawlers only
robotstxt_generate(siteUrl="https://example.com",
                   sitemaps="https://example.com/sitemap.xml",
                   ai={"preset": "training"})

# block the admin area and copy the same rules into the AI groups
robotstxt_generate(disallow="/admin/\n/cart/", ai={"preset": "training"})

# just list the AI crawlers (name, vendor, purpose) without generating
robotstxt_generate(listCrawlers=true)

  


  

    
Other Tools