✓ Free to Use • No Signup • Processed Locally in Your Browser
Build a clean, valid robots.txt: choose which bots the rules apply to, list the paths they may not crawl, add your sitemap, and download the finished file ready to upload to your domain root.
robots.txt is a small text file at yoursite.com/robots.txt that tells well-behaved crawlers which parts of the site they may fetch. It is a crawl management tool — it controls where bots spend their time, so search engines focus on the pages that matter.
Two warnings worth knowing: robots.txt is not a security or privacy tool (blocked pages can still be discovered and, if linked, sometimes indexed without content), and one wrong line — Disallow: / — can remove your whole site from search. Generate carefully, then test in Google Search Console.
A crawler fetches /robots.txt before crawling a site and matches the paths it wants against the rules for its user-agent group. Rules are prefix matches: Disallow: /cart/ blocks everything under /cart/. The wildcard * stands for any characters, and $ marks the end of a URL, so Disallow: /*.pdf$ blocks PDF files specifically. When Allow and Disallow both match, the more specific (longer) rule wins.
If the file is missing, everything is crawlable; if it cannot be fetched due to a server error, well-behaved crawlers pause rather than guess. This generator produces the common, safe patterns — one user-agent group, a list of disallowed paths, optional allows, and the sitemap line that helps engines find your sitemap even before you submit it in Search Console.
Good candidates: admin and login areas, internal search results, cart and checkout flows, print and duplicate-parameter URLs, and staging or test directories. These waste crawl budget and can surface thin or duplicate pages.
Never block the assets pages need to render — CSS, JavaScript and images — because the crawler then sees a broken, unrendered page and judges that. And think twice before blocking anything you actually want indexed: a Disallow line is absolute for that bot, and sites have removed entire sections from search with a single careless rule left over from a development phase. The classic accident is launching with the developer's Disallow: / still in place.
These three are constantly confused because they all influence indexing, but they do different jobs. robots.txt controls crawling — whether the bot may fetch the page at all. A noindex tag controls indexing — the bot may fetch, read and follow links, but must not show the page in results. A canonical tag expresses preference — among duplicate or near-duplicate URLs, this one should represent the set.
The practical consequences: a page blocked in robots.txt can still be indexed (as a link-only result) if other sites link to it, because the bot never fetched it to see a noindex. To keep a crawlable page out of the index, use noindex. To stop wasting crawl on junk URLs, use robots.txt. To consolidate duplicates, use canonicals — and do not combine robots blocks with canonical hopes, since a blocked page's canonical is never read. The Meta Tag Generator includes the robots directive when you need the tag version instead.
For a typical WordPress site, the generated file usually looks like this in spirit: rules for all bots (*), Disallow on /wp-admin/ with an Allow for /wp-admin/admin-ajax.php (some themes and plugins need that file reachable), Disallow on the internal search path (/?s= or /search/), and the Sitemap line pointing at the site’s sitemap index. Nothing else is blocked — content, images and assets stay fully crawlable.
An online shop adds its own lines: /cart/, /checkout/ and account pages disallowed, since those URLs are per-visitor machinery, not content. What neither file does is hide staging copies or secret pages — anything genuinely private belongs behind a login, not behind a robots rule that any curious person can read at yoursite.com/robots.txt. Treat the file as traffic direction for polite crawlers, generate it cleanly, and re-test it after edits.
Privacy: this tool runs entirely in your browser. Nothing you paste is uploaded, stored or tracked — close the tab and it is gone.
At the root of the domain: https://yourdomain.com/robots.txt. It must be a plain text file, publicly reachable, and it applies per subdomain.
Not reliably. A blocked page that has links pointing to it can still appear in results (without a description). To remove a page, use a noindex tag or the removals tool in Search Console instead.
Disallow blocks a path; Allow re-opens a sub-path inside a blocked area — for example blocking /wp-admin/ but allowing /wp-admin/admin-ajax.php, which some themes need crawlable.
That is a policy choice about how your content may be used for AI training and answers. This generator can write rules for a specific bot (choose it as the user-agent) either way.
Tools handle the technical setup. Rankings come from authority — browse 1,502 real publisher sites with live metrics and fixed prices, or see our packages.