✓ Free to Use • No Signup • Live Check — We Fetch the URL for You
Enter a domain, pick a crawler and type a path. The tool fetches the site’s live robots.txt, applies its rules the way crawlers do — matching group, longest rule wins — and tells you Allowed or Blocked, naming the exact rule that decided it. The full file and any sitemap declarations are shown underneath.
robots.txt is the small file at yoursite.com/robots.txt that tells well-behaved crawlers which paths they may fetch. It is a crawl-management tool, not a security tool: a blocked page can still be discovered through links and can sometimes be indexed without its content.
Testing matters because the failures are silent. One stray line — Disallow: / in the wrong group — can keep a whole section, or a whole site, out of search, and nothing in a browser looks broken. This tester reads the live file, not a copy, so what you see is what a crawler sees today. If you need to write or repair a file, the Robots.txt Generator builds one from your rules.
A robots.txt file is organised into groups. Each group names one or more crawlers in User-agent lines, followed by Allow and Disallow rules that apply only to those crawlers; a group built for one bot says nothing about the rest. Rules match against the path of a URL, starting from the first character after the domain. An empty Disallow line blocks nothing, while a Disallow of the root path blocks the entire site for that crawler. The asterisk acts as a wildcard that stands for any sequence of characters, which is how one rule can cover a whole family of similar URLs.
When several rules match the same path, the longest matching rule wins, measured by the length of the path pattern rather than by the order lines appear in the file. This is why a specific Allow can open one file inside an otherwise blocked folder, and why a short, broad Disallow often overrules the careful exceptions people add above it. Matching is also case-sensitive, so a rule written for one capitalisation may leave the differently cased version of the same path fully open. The tester applies exactly this logic for the crawler and path you choose, then shows you the single rule that decided the verdict.
The classic accident is a site launch. Rules written for a staging copy, where blocking everything was the point, get carried into the live file and quietly tell search engines to stay away from the real site. A close second is the over-broad block: someone bars a single folder such as a search results path and, with one prefix, also bars the product or article sections that happen to share its opening characters. The tester shows the verdict for the exact path you care about, so these errors surface in seconds rather than in a traffic report weeks later.
Other traps are quieter. A file written years ago names the crawlers of its era, so newer agents, including the AI crawlers many sites now want to manage, may fall under no group or under a catch-all that was never meant for them. Syntax slips matter too: a malformed line can be skipped by parsers, changing what the file says. And the Sitemap lines in the file are easy to lose in an edit, which removes the discovery hint every crawler reads. Running each important path and crawler pairing through the tester before you publish turns all of these from silent risks into a checklist.
Privacy: to run this check, our server fetches the public URL you enter — the same request any browser or crawler makes. The URL and its results are not stored; close the tab and they are gone.
When an Allow and a Disallow rule both match a path, the longest (most specific) pattern wins — that is the standard behaviour crawlers follow. The tester shows you the winning rule so you can see the logic, not just the outcome.
The fetch returns a 404 and every crawler treats the site as fully crawlable. That is a valid state — the tester will say so and test the path against an empty rule set.
Not reliably. It blocks fetching, not indexing: a blocked URL with links pointing to it can appear in results without a description. To keep a page out of the index, use a noindex directive on the page itself — check it with the <a href="/seo-tools/meta-tags-checker/">Meta Tags Checker</a>.
Google’s crawlers follow robots.txt rules for the paths they fetch, with the documented exception of some product-specific crawlers. Other bots vary — which is why the file is a request system, not a lock.
Tools handle the technical setup. Rankings come from authority — browse 1,502 real publisher sites with live metrics and fixed prices, or see our packages.