Robots.txt Generator
Compose a valid robots.txt with allow/disallow rules, crawl-delay and a sitemap line.
SEO Tools
RULES → robots.txt
Loading tool…
Files are processed in your browser. Nothing is uploaded to a server.
The robots.txt file sits at the root of your domain and tells search-engine crawlers which parts of your site they may or may not request. A single typo can accidentally hide your whole site from Google, so it pays to generate it carefully.
This tool builds a syntactically correct file from simple inputs: choose which folders to block, set an optional crawl delay, and add your sitemap URL.
How to build your robots.txt
- Choose whether to allow all crawlers or target a specific user-agent.
- Add the paths you want to Disallow (for example
/wp-admin/or/cart/). - Optionally add explicit Allow rules and a crawl-delay value.
- Paste your sitemap URL, then copy the file and upload it to your site root as
robots.txt.
Common directives explained
| Directive | Meaning | Example |
|---|---|---|
| User-agent | Which crawler the rules apply to | User-agent: * |
| Disallow | Path the crawler should not request | Disallow: /private/ |
| Allow | Exception inside a disallowed path | Allow: /private/public.html |
| Sitemap | Absolute URL of your XML sitemap | Sitemap: https://site.com/sitemap.xml |
| Crawl-delay | Seconds between requests (not honored by Google) | Crawl-delay: 10 |
Built-in safeguards
- Presets for blocking common admin, cart and search paths.
- Reminds you that
Disallow: /blocks the entire site. - Adds the sitemap directive so crawlers can find your URLs faster.
- Outputs plain text ready to save as robots.txt.
Frequently asked questions
Does robots.txt keep pages out of Google?
Not reliably. It stops crawling, but a blocked URL can still be indexed if linked elsewhere. To truly hide a page, use a
noindex meta tag and allow crawling.Where must robots.txt live?
At the very root of your domain, reachable at https://yourdomain.com/robots.txt. It will not work in a subfolder.
Is crawl-delay respected?
Bing and some crawlers honor it; Googlebot ignores it. Set crawl rate for Google in Search Console instead.