Robots.txt Generator

Generate robots.txt files for your website to control search engine crawling. Create SEO-optimized robots.txt files easily.

User Agent Select search engine bot

Custom Agent If custom agent selected

Crawl Delay Delay between requests

Custom Delay If custom delay selected

Disallow Paths Paths to block from crawling

Allow Paths Paths to explicitly allow

Sitemap URL URL to your sitemap (optional)

Total Rules

User Agent

File Size

Generated

Advertisement Space

300x250

User agent control

Path allow/disallow

Crawl delay settings

Sitemap integration

SEO compliant

User-agent: * (all bots)

Disallow: Block paths

Allow: Explicit allow

Place in root directory

User-Agent

Disallow

Allow

Crawl-Delay

Sitemap

Control