Robotstxt Generator
Generate a robots.txt file with crawl directives for search engine bots. Enter values for instant results with step-by-step formulas.
Reviewed by Daniel Agrici, Founder & Lead Developer
Formula
User-agent → Disallow/Allow → Crawl-delay → Sitemap
A robots.txt file uses directives to instruct crawlers. User-agent specifies which bot, Disallow blocks paths, Allow overrides within blocked directories, Crawl-delay sets request intervals, and Sitemap points to your XML sitemap.
Worked Examples
Example 1: Standard Business Website
Problem:Generate a robots.txt for a business site that blocks admin, login, and staging areas while providing the sitemap location.
Solution:User-agent: *\nDisallow: /admin/\nDisallow: /login/\nDisallow: /staging/\nDisallow: /api/\nAllow: /api/public/\n\nSitemap: https://example.com/sitemap.xml
Result:Clean robots.txt with 4 disallow rules, 1 allow override, and sitemap
Example 2: Blog Blocking AI Crawlers
Problem:Create a robots.txt for a blog that allows all search engines but blocks AI training crawlers.
Solution:User-agent: *\nDisallow: /draft/\nDisallow: /preview/\n\nUser-agent: GPTBot\nDisallow: /\n\nUser-agent: Google-Extended\nDisallow: /\n\nUser-agent: CCBot\nDisallow: /\n\nSitemap: https://blog.example.com/sitemap.xml
Result:Search engines can crawl freely; AI training bots are fully blocked
References
Reviewed by Daniel Agrici, Founder & Lead Developer · Editorial policy