Robots.txt generator

Build crawler groups and allow/disallow rules with a live robots.txt preview and presets.

Live robots.txt94 characters

User-agent: *
Disallow: /admin/
Allow: /admin/help/

Sitemap: https://example.com/sitemap.xml

1Crawler groups

2Rules

Crawling rules are public instructions, not access control. An empty Disallow permits all paths. A specific crawler group is selected before the * fallback; rules from separate matching groups may combine.

Presets replace the groups below.

Sitemap URLs (one per line)Optional. Up to 10 absolute HTTP(S) sitemap URLs.

Group 1

User agents for group 1Comma-separated product tokens, for example Googlebot, Bingbot. Use * for other crawlers.

Rule 1.1

Path 1.1

Rule 1.2

Path 1.2

Processed on your device

How it works & useful details

How this tool works

Place robots.txt at the site origin root. Supports RFC 9309 User-agent, Allow and Disallow plus Sitemap URL lines; crawl-delay and crawler simulation are not included. Product tokens allow ASCII letters, hyphens and underscores. Paths start with /; * and $ are preserved. Percent-encode spaces and #. Empty rules impose no restriction. Crawlers may ignore rules; this file does not protect private pages or guarantee deindexing. Up to 8 groups, 20 rules per group, 100 rules total and 10 sitemap URLs.

RFC 9309: Robots Exclusion Protocol