Where do I place the robots.txt file?

It must be in the root of your domain, e.g. https://example.com/robots.txt.

Does robots.txt block all bots?

No. robots.txt is advisory. Malicious bots may ignore it. Use server-level access controls to truly block traffic.

Can I allow some paths and disallow others?

Yes. Add multiple Allow and Disallow rules per User-agent. More specific rules generally take precedence.

Robots.txt Generator

Build a robots.txt file with rules, sitemap, and common presets.

rules

Rule 1

User-agent

AllowDisallow

Sitemap URL (optional)

output

User-agent: *

Related Tools

...Sitemap Generator

Generate an XML sitemap from a list of URLs.

<>Meta Tag Generator

Generate HTML meta tags including Open Graph and Twitter Card.

FAQ

Where do I place the robots.txt file?: It must be in the root of your domain, e.g. https://example.com/robots.txt.
Does robots.txt block all bots?: No. robots.txt is advisory. Malicious bots may ignore it. Use server-level access controls to truly block traffic.
Can I allow some paths and disallow others?: Yes. Add multiple Allow and Disallow rules per User-agent. More specific rules generally take precedence.

Create a valid robots.txt file by adding User-agent rules, Allow and Disallow directives, and a Sitemap URL. Quick presets cover the most common setups: Allow All, Block All, and Block Specific Bots. The generated file is shown with syntax highlighting and a copy button.