Robots.txt Generator
Configure crawler rules for your site
Rules
Configure how search engines should crawl your site
Preview
Generated robots.txt file
# Generated with Robots.txt Generator
# Default rule for all bots
User-agent: *
Disallow: /admin/
Disallow: /private/Generate a robots.txt file to control how search engines crawl your website. Add rules for different crawlers and include sitemap references.