Robots.txt Generator
Build a valid robots.txt file in seconds. Add rules for any user-agent, block directories, set crawl delays, and add your sitemap URL — with live syntax-highlighted preview and validation.
What is robots.txt?
robots.txt is a plain text file placed at the root of your website that tells search engine crawlers and other bots which pages or sections they are allowed or not allowed to crawl. It follows the Robots Exclusion Standard — a protocol all major search engines respect by convention.
When a bot visits your site, it first checks yourdomain.com/robots.txt before crawling anything else. If the file contains rules that apply to that bot, it will follow them. If no file exists, bots treat the entire site as open for crawling.
robots.txt Syntax Guide
- User-agent: specifies which bot the following rules apply to. Use
*for all bots. - Disallow: tells bots not to crawl a path.
Disallow: /admin/blocks everything under /admin/. - Allow: overrides a Disallow — useful when you want to block a directory but allow a specific page within it.
- Crawl-delay: asks bots to wait N seconds between requests. Useful for protecting server load (note: Google ignores this).
- Sitemap: points to your sitemap.xml. Can be listed at the bottom of the file regardless of user-agent context.
Frequently Asked Questions
Disallow stops Google from crawling it, but Google can still index a URL it has never visited if it finds links to it on other pages. To prevent a page from appearing in search results entirely, use a noindex meta tag or X-Robots-Tag HTTP header on the page itself. robots.txt and noindex serve different purposes and are often used together.Disallow: / for their user-agents will stop them from training on your content. However, blocking AI crawlers has no impact on your Google rankings and doesn't prevent use of content that was already crawled before you added the rules. Many publishers choose to block them as a matter of principle or licensing.User-agent: * section with broad rules, and then separate sections for specific bots like Googlebot or Bingbot with different rules. When a bot reads the file, it follows the most specific section that applies to it, then falls back to * rules. Most bots stop at the first matching group and ignore other groups.Enjoying this tool?
Every tool here is free, forever — no signup, no paywall. If this one just saved you real time, a small tip keeps them free and ad-light.
☕ Support This Tool →Need something built, not just formatted?
This toolbox is free. If your business needs custom software, AI integration, or a system built around this kind of tooling, that's what I do for a living.