Skip to content
littlekit.app

Robots.txt generator

Build crawler groups and allow/disallow rules with a live robots.txt preview and presets.

Live robots.txt94 characters
User-agent: *
Disallow: /admin/
Allow: /admin/help/

Sitemap: https://example.com/sitemap.xml
1Crawler groups
2Rules

Crawling rules are public instructions, not access control. An empty Disallow permits all paths. A specific crawler group is selected before the * fallback; rules from separate matching groups may combine.

Presets replace the groups below.

Optional. Up to 10 absolute HTTP(S) sitemap URLs.

Group 1

Comma-separated product tokens, for example Googlebot, Bingbot. Use * for other crawlers.
Up to 8 groups, 20 rules per group, 100 rules total.
Processed on your device

How it works & useful details

How this tool works

Place robots.txt at the site origin root. Supports RFC 9309 User-agent, Allow and Disallow plus Sitemap URL lines; crawl-delay and crawler simulation are not included. Product tokens allow ASCII letters, hyphens and underscores. Paths start with /; * and $ are preserved. Percent-encode spaces and #. Empty rules impose no restriction. Crawlers may ignore rules; this file does not protect private pages or guarantee deindexing. Up to 8 groups, 20 rules per group, 100 rules total and 10 sitemap URLs.

RFC 9309: Robots Exclusion Protocol

About this tool

Robots.txt generator builds crawler groups and allow or disallow rules with a live preview of the resulting file. Presets cover the common cases, and any rule can be edited directly.

The output follows the robots.txt conventions, including wildcard and end-anchor patterns in supported positions. Hosting the file at your site root is up to you; the tool only writes the text.

How to use it

  1. Start from a preset or add crawler groups manually.
  2. Add allow and disallow rules, including wildcards where supported.
  3. Review the live preview of the file.
  4. Copy or download the robots.txt content.

Worked example

User-agent: *
Disallow: /admin/
Allow: /admin/help/

Sitemap: https://example.com/sitemap.xml

This asks all crawlers to stay out of /admin/ except the public help pages under /admin/help/. Under RFC 9309, the most specific (longest) matching rule wins, so the Allow line takes precedence for help pages. The Sitemap line tells crawlers where to find the list of pages you want indexed.

What robots.txt can and cannot do

robots.txt must live at the root of the host, as in https://example.com/robots.txt. It controls crawling, not indexing or access. A disallowed URL can still appear in search results if other sites link to it, and anyone can still open it. To keep a page out of search results, allow crawling and use a noindex meta tag. To protect private content, require a login.

  • Disallow: / with User-agent: * blocks all compliant crawlers from the whole site, a common accident on staging sites that go live.
  • An empty Disallow: line allows everything.
  • * matches any sequence of characters, and $ anchors the end of the URL, as in Disallow: /*.pdf$.
  • Groups for specific crawlers, such as User-agent: GPTBot, override the * group for that crawler.

Frequently asked questions

Does rules order matter?

Yes. More specific patterns take precedence in major crawlers, and the preview keeps your rule order so you can reason about the result.

Can robots.txt block a page from search results?

Not reliably. A robots.txt rule prevents crawling, but an indexed page can still appear in results; use a robots meta tag to control indexing.

What is the limit?

Up to 8 crawler groups, 20 rules per group, 100 rules in total, and 10 sitemap URLs. That covers the robots.txt files of almost all sites.

Is Crawl-delay supported?

It is not part of RFC 9309, and Google ignores it, so the generator does not write it. Some other crawlers honour it. Add the line manually if you need it.