Robots.txt Tester
Check whether a specific URL is allowed or blocked by your robots.txt before search engines crawl it. Enter the file, a URL, and a user-agent, and see exactly which rule wins and why.
How to test a URL against robots.txt
- Paste the full contents of your robots.txt into the first box.
- Enter the URL or path you want to check and the crawler user-agent.
- Read the allowed or blocked badge and the matching rule it reports.
Examples
A longer Allow overrides a broad Disallow
robots.txt: User-agent: *\nDisallow: /folder\nAllow: /folder/public URL: /folder/public/page.html
Allowed by the most specific matching rule "Allow: /folder/public".
Frequently asked questions
How does the tester decide which rule wins?
It follows Google's rule precedence: the most specific rule wins, where specificity is the number of characters in the rule path. The longest matching Allow or Disallow decides. When an Allow and a Disallow match the same length, Allow wins.
How are User-agent groups chosen?
Crawlers obey the most specific group that names them. An exact or matching user-agent token (like Googlebot) overrides the catch-all star group, and only that one group's rules are evaluated. Unlisted bots fall back to the star group.
What do the * wildcard and $ anchor mean?
An asterisk matches any run of characters, so Disallow: /*.pdf blocks every PDF path. A dollar sign at the end anchors the match to the end of the URL, so Disallow: /*.php$ blocks /index.php but not /index.php?id=1.
Does an empty Disallow block anything?
No. A bare Disallow: line with no path means allow everything for that group. It is the standard way to grant a crawler full access to your site.
Does robots.txt keep a page out of Google's index?
No. It only controls crawling. A blocked URL can still be indexed if other pages link to it. To keep a page out of search results, allow crawling and add a noindex meta tag, or use HTTP authentication.
Is my robots.txt or URL sent to a server?
No. The test runs entirely in your browser. Nothing you paste is uploaded, logged, or stored anywhere.
Related tools
Robots.txt Generator
Create a valid robots.txt file in seconds. Pick allow all, block all, or custom rules, add your sitemap URL, then copy or download the file.
XML Sitemap Generator
Paste your URLs and get a valid XML sitemap in seconds. Set change frequency and priority, then copy or download sitemap.xml for Search Console.
Canonical Tag Generator
Generate a clean rel=canonical link tag from any URL. Strip tracking params, force HTTPS and lowercase the host, then copy the tag. Runs in your browser.
Article Schema Generator
Generate Article, NewsArticle or BlogPosting JSON-LD schema. Add headline, author, publisher and dates, then copy the script into your page.
Breadcrumb Schema Generator
Build BreadcrumbList JSON-LD structured data. Add your breadcrumb trail, reorder the steps, then copy or download the script for your page.
Bulk Hreflang Generator
Generate hreflang annotations for many languages at once. Add language and URL rows, pick HTML, XML sitemap, or HTTP header output. Runs in your browser.