Robotstxt Generator
Generate a properly formatted robots.txt file with crawl directives for search engine bots. Block AI crawlers, set crawl delays, and specify sitemaps.
Robotstxt Generator
Calculator
Adjust values & calculateEnter your values below. Every result is computed in your browser — no data is sent to any server.
Formula: User-agent → Disallow/Allow → Crawl-delay → Sitemap
Worked example — Clean robots.txt with 4 disallow rules, 1 allow override, and sitemap
Formula
User-agent → Disallow/Allow → Crawl-delay → Sitemap
A robots.txt file uses directives to instruct crawlers. User-agent specifies which bot, Disallow blocks paths, Allow overrides within blocked directories, Crawl-delay sets request intervals, and Sitemap points to your XML sitemap.
Worked Examples
Example 1: Standard Business Website
Problem:Generate a robots.txt for a business site that blocks admin, login, and staging areas while providing the sitemap location.
Solution:User-agent: * Disallow: /admin/ Disallow: /login/ Disallow: /staging/ Disallow: /api/ Allow: /api/public/ Sitemap: https://example.com/sitemap.xml
Result:Clean robots.txt with 4 disallow rules, 1 allow override, and sitemap
Example 2: Blog Blocking AI Crawlers
Problem:Create a robots.txt for a blog that allows all search engines but blocks AI training crawlers.
Solution:User-agent: * Disallow: /draft/ Disallow: /preview/ User-agent: GPTBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: CCBot Disallow: / Sitemap: https://blog.example.com/sitemap.xml
Result:Search engines can crawl freely; AI training bots are fully blocked
Frequently Asked Questions
What is a robots.txt file and how does it work?
A robots.txt file is a plain text file placed in the root directory of a website that provides instructions to web crawlers and search engine bots about which pages or sections they may or may not access. It follows the Robots Exclusion Protocol, first proposed by Martijn Koster in 1994. When a crawler visits your site, it first checks for a robots.txt file at yourdomain.com/robots.txt before crawling any pages. The file uses directives like User-agent to specify which bots the rules apply to, Disallow to block access to paths, Allow to permit specific paths within blocked directories, and Sitemap to point crawlers to your XML sitemap. It is important to note that robots.txt is advisory, not enforceable.
Does robots.txt prevent pages from appearing in search results?
No, robots.txt does not guarantee that pages will be excluded from search results. While it tells well-behaved crawlers not to access certain pages, it does not remove pages already indexed. If other sites link to a blocked page, search engines may still list it in results with a note that the description is unavailable because of robots.txt restrictions. To truly prevent indexing, you should use the noindex meta tag or X-Robots-Tag HTTP header instead. Robots.txt is best used to manage crawl budget by preventing crawlers from accessing duplicate content, admin panels, staging environments, and resource-heavy pages that do not need to appear in search results.
What is crawl-delay and how should I set it?
The Crawl-delay directive tells crawlers to wait a specified number of seconds between successive requests to your server. This is useful for smaller servers that cannot handle rapid, successive requests from aggressive crawlers. For example, setting Crawl-delay to 10 means the bot should wait 10 seconds between each page request. Google does not officially support the Crawl-delay directive and instead recommends adjusting crawl rate through Google Search Console. However, Bing, Yandex, and many other crawlers do respect it. Be cautious with high values as they significantly slow down indexing. A value of 1 to 5 seconds is typically sufficient for most small to medium-sized websites.
How do I block AI crawlers like GPTBot and Google-Extended?
To block AI training crawlers, add specific User-agent directives to your robots.txt file. GPTBot is OpenAI's crawler used for training data, ChatGPT-User is the browsing plugin crawler, Google-Extended is Google's AI training crawler separate from Googlebot, CCBot is Common Crawl's bot used by many AI companies, and ClaudeBot is Anthropic's crawler. To block them, add entries like 'User-agent: GPTBot' followed by 'Disallow: /' for each bot. This prevents these crawlers from accessing your content for AI model training while still allowing regular search engine indexing. This generator includes a convenient toggle to block all major AI crawlers at once.
What are common mistakes to avoid in robots.txt?
Several common mistakes can harm your SEO when configuring robots.txt. First, accidentally blocking your entire site with 'Disallow: /' under 'User-agent: *' will prevent all crawlers from indexing any content. Second, blocking CSS, JavaScript, or image files that search engines need to render pages properly can hurt your rankings, as Google needs to render pages to evaluate content quality. Third, using robots.txt to hide sensitive content instead of proper authentication provides no real security since anyone can read the file. Fourth, not including a Sitemap directive misses an opportunity to help crawlers discover your content efficiently. Fifth, forgetting that robots.txt is case-sensitive for paths means /Admin/ and /admin/ are treated as different directories.
How do crawlers resolve a conflict between an Allow and a Disallow rule?
Google evaluates every matching rule and applies the one with the longest path match, counting wildcard characters as literal length, rather than taking whichever appears first in the file. When two matching rules are the same length, the least restrictive one wins, so the Allow prevails. That is why an Allow of /admin/public/ successfully carves an exception out of a broader Disallow of /admin/. Not every crawler implements this logic, so keep exception rules simple and few.
How do the wildcard and dollar-sign characters work in robots.txt paths?
An asterisk matches any sequence of characters, so Disallow: /*?sessionid= blocks that parameter wherever it appears in the query string. A dollar sign anchors the pattern to the end of the URL, so Disallow: /*.pdf$ blocks PDF files but not a URL that merely contains .pdf mid-path. Paths without either character are treated as prefixes, which means Disallow: /tmp also blocks /tmpfiles.html unless you write the trailing slash.
Does each subdomain and protocol need its own robots.txt file?
Yes. A robots.txt file governs exactly one origin, meaning the combination of scheme, host and port, and it must sit at the root of that origin. Rules at https://example.com/robots.txt do not apply to blog.example.com, to a country subdomain, or to a service on a non-standard port, and a file placed in a subdirectory is ignored entirely. Sitemap directives are the exception: they may point at fully qualified URLs on other hosts.
What do crawlers do when robots.txt returns a 404 or a server error?
A 404 is read as no restrictions, so crawlers treat the whole site as crawlable, which is the safe default for sites that never needed a file. A 5xx response is different: Google treats a server error as a temporary signal to hold off crawling, and prolonged errors can slow discovery across the site. Serve the file as static plain text rather than generating it from your application, so an outage cannot take your crawl directives down with it.
Why can a Disallow rule stop a noindex tag from ever taking effect?
A noindex directive lives in the page's HTML head or in an X-Robots-Tag response header, and a crawler can only act on it by fetching the page. If robots.txt disallows that URL, the crawler never fetches it, never sees the noindex, and the URL can still be listed from external links with no description. To remove a page from search, leave it crawlable and serve noindex, then add the Disallow only once the URL has dropped out of the index.
References
Reviewed for accuracy by Daniel Agrici, Founder & Lead Developer · Editorial policy
Related Calculators
🧮Canonical URL Generator
Generate canonical URL tags to prevent duplicate content issues in search engines.
🧮Schema Markup Generator
Generate JSON-LD structured data markup for articles, products, FAQs, and recipes.
🧮Hreflang Tag Generator
Generate hreflang tags for multilingual and multi-regional websites.
🧮Open Graph Tag Generator
Generate Open Graph and Twitter Card meta tags for social media sharing optimization.
🧮Marketing ROI Calculator
Calculate marketing campaign return on investment from spend and revenue generated.
🧮Content Marketing ROI Calculator
Calculate the ROI of content marketing efforts from production costs and generated revenue.
🧮Domain Authority Checker
Check and compare domain authority scores for competitive SEO analysis.
🧮Keyword Density Checker
Analyze keyword density and frequency distribution in any webpage or text content.