📈 Web & SEO ✅ 100% Free ⚡ Instant

Robots.txt Tester

Test whether any URL is allowed or blocked by your robots.txt rules. Paste your robots.txt content, specify a user agent, and check multiple URLs at once. Highlights the matching rule.

robots.txt Content

User Agent to Test

URL to Test (one per line)

Paste robots.txt and enter URLs to test.

📋 Parsed Rules

Paste a robots.txt above to see parsed rules.

What is robots.txt?

The robots.txt file is a text file placed at the root of a website (/robots.txt) that tells search engine crawlers which pages or sections of a site to crawl or avoid. It uses the Robots Exclusion Protocol — a set of rules defined by User-agent (which bot the rule applies to) and Disallow/Allow directives (which paths to block or permit). It is not a security measure — it is a crawling guideline that well-behaved bots follow voluntarily.

Frequently Asked Questions

Not directly. Disallowing a URL prevents crawlers from accessing it, but if other pages link to it, search engines may still index it without crawling. To prevent indexing, use a noindex meta tag or X-Robots-Tag HTTP header instead.

Disallow: / blocks the entire site for that user agent. An empty Disallow: (or omitting it entirely) allows full access — it explicitly permits crawling everything. Do not confuse the two.

When both Allow and Disallow apply to the same URL, the more specific rule wins. If equal specificity, Allow takes precedence (in Google's implementation). For example, Disallow: /private/ and Allow: /private/public/ — the Allow wins for /private/public/ because it is more specific.

Copied!