Where does robots.txt go?

It must live at the root of your domain, served as text at https://example.com/robots.txt. Crawlers only read it from that exact location, not from subfolders.

What does Disallow do?

Disallow tells compliant crawlers not to fetch URLs matching the given path prefix. An empty Disallow value means nothing is blocked, allowing the whole site.

Does robots.txt guarantee privacy?

No. It is advisory and only respected by well-behaved bots. Sensitive pages should be protected by authentication, not by Disallow, since the file itself is public.

Crawl-delay asks a crawler to wait the given number of seconds between requests. Google ignores it, but Bing and some other crawlers honor it to reduce server load.

Should I include a Sitemap line?

Yes. A Sitemap directive pointing to your sitemap URL helps search engines discover all your pages. It uses an absolute URL and can appear once per file.

robots.txt Builder — Gera Tools

Email me this result

Get this tool's output sent to your inbox, plus one useful tool a week. No spam, unsubscribe any time.

robots.txt, built correctly

robots.txt is the file crawlers read first to learn which parts of your site they may fetch. A small mistake — a stray Disallow: / or a malformed path — can deindex an entire site, so it pays to generate it cleanly. This builder produces a valid file from simple inputs and lets you copy or download it.

How it works

The Robots Exclusion Protocol groups directives under one or more User-agent lines. This builder emits a single group: a User-agent line, then one Disallow: line per disallow path and one Allow: line per allow path. Paths are matched as prefixes from the site root, so Disallow: /admin blocks /admin and everything beneath it. An optional Crawl-delay line is added when you set a positive number. Finally a Sitemap: directive with your absolute sitemap URL is appended at the end, which applies regardless of user-agent group.

Tips and example

To allow the whole site, leave the disallow box empty. A generated file looks like this:

User-agent: *
Disallow: /admin
Disallow: /cart
Allow: /admin/help
Crawl-delay: 10
Sitemap: https://example.com/sitemap.xml

Always verify the result in Google Search Console’s robots.txt tester before deploying — and remember that blocking a URL here does not remove it from the index if it is already cached or linked elsewhere.