Generate a robots.txt File
Generate a clean robots.txt file with sitemap and crawl rules.
How to use the Robots.txt Generator
Choose the user agent
Use * for every crawler, or name a specific one when a rule should only apply to it.
Set your allow and disallow paths
List the paths crawlers should skip, such as admin or internal search pages, and any exceptions to allow.
Add your sitemap and copy
Enter your sitemap URL, then copy the file and save it as robots.txt at the root of your domain.
About the Robots.txt Generator
robots.txt is the first file a crawler asks for and one of the easiest to get catastrophically wrong. It sits at the root of your domain and tells crawlers which paths they may request. A stray slash in a disallow rule can hide an entire site from search, and because nothing visibly breaks, that mistake often survives for months before anyone connects it to the traffic that never arrived.
The syntax is small but unforgiving: rules are matched by prefix, order and specificity interact, and a blank or malformed file is treated very differently from a permissive one. This tool builds a valid file from the parts you actually care about — which crawler a rule applies to, what to allow, what to disallow, and where your sitemap lives.
One thing worth being clear about: robots.txt controls crawling, not indexing or access. A disallowed URL can still appear in results if other sites link to it, and the file is public, so listing a secret path advertises it. Anything genuinely private needs authentication or a noindex tag, not a line in this file. Use it to keep crawlers away from admin paths, search result pages and duplicate parameter URLs, and to point them at your sitemap.
Frequently asked questions
- Where do I put the robots.txt file?
- At the root of your domain, so it is reachable at yoursite.com/robots.txt. Crawlers do not look anywhere else.
- Does robots.txt stop a page being indexed?
- No. It stops crawling, but a blocked URL can still be listed if other sites link to it. Use a noindex tag to keep a page out of the index.
- Should I list my sitemap in robots.txt?
- Yes. It is the standard way to point every crawler at your sitemap without submitting it to each one individually.