Simple.Tools -> Robots.txt Generator
Create a clean robots.txt file with common crawl directives and sitemap entries.
User-Agent
Crawl-Delay (seconds, optional)
Disallow Paths (one per line)
/admin/ /private/ /temp/
Allow Paths (one per line)
/assets/
Sitemap URL (optional)
Generate robots.txt
Copy Content
Reset
Clear