Robotstxt Generator
Generate a robots.txt file with crawl directives for search engine bots. Enter values for instant results with step-by-step formulas.
Reviewed for accuracy by Daniel Agrici, Founder & Lead Developer
Robotstxt Generator
Calculator
Adjust values & calculateEnter your values below. Every result is computed in your browser — no data is sent to any server.
Formula: User-agent → Disallow/Allow → Crawl-delay → Sitemap
Worked example — Clean robots.txt with 4 disallow rules, 1 allow override, and sitemap
Formula
User-agent → Disallow/Allow → Crawl-delay → Sitemap
A robots.txt file uses directives to instruct crawlers. User-agent specifies which bot, Disallow blocks paths, Allow overrides within blocked directories, Crawl-delay sets request intervals, and Sitemap points to your XML sitemap.
Worked Examples
Example 1: Standard Business Website
Problem:Generate a robots.txt for a business site that blocks admin, login, and staging areas while providing the sitemap location.
Solution:User-agent: * Disallow: /admin/ Disallow: /login/ Disallow: /staging/ Disallow: /api/ Allow: /api/public/ Sitemap: https://example.com/sitemap.xml
Result:Clean robots.txt with 4 disallow rules, 1 allow override, and sitemap
Example 2: Blog Blocking AI Crawlers
Problem:Create a robots.txt for a blog that allows all search engines but blocks AI training crawlers.
Solution:User-agent: * Disallow: /draft/ Disallow: /preview/ User-agent: GPTBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: CCBot Disallow: / Sitemap: https://blog.example.com/sitemap.xml
Result:Search engines can crawl freely; AI training bots are fully blocked
Frequently Asked Questions
What is a robots.txt file and how does it work?
A robots.txt file is a plain text file placed in the root directory of a website that provides instructions to web crawlers and search engine bots about which pages or sections they may or may not access. It follows the Robots Exclusion Protocol, first proposed by Martijn Koster in 1994. When a crawler visits your site, it first checks for a robots.txt file at yourdomain.com/robots.txt before crawling any pages. The file uses directives like User-agent to specify which bots the rules apply to, Disallow to block access to paths, Allow to permit specific paths within blocked directories, and Sitemap to point crawlers to your XML sitemap. It is important to note that robots.txt is advisory, not enforceable.
Does robots.txt prevent pages from appearing in search results?
No, robots.txt does not guarantee that pages will be excluded from search results. While it tells well-behaved crawlers not to access certain pages, it does not remove pages already indexed. If other sites link to a blocked page, search engines may still list it in results with a note that the description is unavailable because of robots.txt restrictions. To truly prevent indexing, you should use the noindex meta tag or X-Robots-Tag HTTP header instead. Robots.txt is best used to manage crawl budget by preventing crawlers from accessing duplicate content, admin panels, staging environments, and resource-heavy pages that do not need to appear in search results.
What is crawl-delay and how should I set it?
The Crawl-delay directive tells crawlers to wait a specified number of seconds between successive requests to your server. This is useful for smaller servers that cannot handle rapid, successive requests from aggressive crawlers. For example, setting Crawl-delay to 10 means the bot should wait 10 seconds between each page request. Google does not officially support the Crawl-delay directive and instead recommends adjusting crawl rate through Google Search Console. However, Bing, Yandex, and many other crawlers do respect it. Be cautious with high values as they significantly slow down indexing. A value of 1 to 5 seconds is typically sufficient for most small to medium-sized websites.
How do I block AI crawlers like GPTBot and Google-Extended?
To block AI training crawlers, add specific User-agent directives to your robots.txt file. GPTBot is OpenAI's crawler used for training data, ChatGPT-User is the browsing plugin crawler, Google-Extended is Google's AI training crawler separate from Googlebot, CCBot is Common Crawl's bot used by many AI companies, and ClaudeBot is Anthropic's crawler. To block them, add entries like 'User-agent: GPTBot' followed by 'Disallow: /' for each bot. This prevents these crawlers from accessing your content for AI model training while still allowing regular search engine indexing. This generator includes a convenient toggle to block all major AI crawlers at once.
What are common mistakes to avoid in robots.txt?
Several common mistakes can harm your SEO when configuring robots.txt. First, accidentally blocking your entire site with 'Disallow: /' under 'User-agent: *' will prevent all crawlers from indexing any content. Second, blocking CSS, JavaScript, or image files that search engines need to render pages properly can hurt your rankings, as Google needs to render pages to evaluate content quality. Third, using robots.txt to hide sensitive content instead of proper authentication provides no real security since anyone can read the file. Fourth, not including a Sitemap directive misses an opportunity to help crawlers discover your content efficiently. Fifth, forgetting that robots.txt is case-sensitive for paths means /Admin/ and /admin/ are treated as different directories.
References
Reviewed for accuracy by Daniel Agrici, Founder & Lead Developer · Editorial policy
Related Calculators
🧮Canonical URL Generator
Generate canonical URL tags to prevent duplicate content issues in search engines.
🧮Schema Markup Generator
Generate JSON-LD structured data markup for articles, products, FAQs, and recipes.
🧮Hreflang Tag Generator
Generate hreflang tags for multilingual and multi-regional websites.
🧮Open Graph Tag Generator
Generate Open Graph and Twitter Card meta tags for social media sharing optimization.
🧮Marketing ROI Calculator
Calculate marketing campaign return on investment from spend and revenue generated.
🧮Content Marketing ROI Calculator
Calculate the ROI of content marketing efforts from production costs and generated revenue.
🧮Domain Authority Checker
Check and compare domain authority scores for competitive SEO analysis.
🧮Keyword Density Checker
Analyze keyword density and frequency distribution in any webpage or text content.