Skip to content
littlekit.app
← All guidesGUIDE · UPDATED 2026-09-24

robots.txt, sitemaps, and meta tags: what search engines read

What each file and tag controls, why blocking in robots.txt does not remove pages from results, and a launch checklist.

Search engines read a handful of small files and tags to decide what to crawl, what to index, and how to show your pages. Each one controls something specific, and the most damaging SEO mistakes come from expecting one to do another’s job. The classic example is blocking a page in robots.txt and wondering why it still appears in results. This guide explains what robots.txt, sitemaps, and meta tags each do.

robots.txt controls crawling

robots.txt lives at the root of a host, such as https://example.com/robots.txt, and asks crawlers not to fetch certain paths. A typical file:

User-agent: *
Disallow: /admin/
Allow: /admin/help/

Sitemap: https://example.com/sitemap.xml

Rules are grouped by User-agent. Under RFC 9309, the standard that formalized the protocol in 2022, the most specific (longest) matching rule wins, which is why the Allow line keeps the help pages crawlable. * matches any sequence of characters, and $ anchors the end of a URL, as in Disallow: /*.pdf$. The robots.txt generator builds groups and rules with a live preview.

What robots.txt does not do:

  • It does not prevent indexing. A disallowed URL can still appear in search results, without a description, if other sites link to it. To keep a page out of results, allow crawling and add a noindex robots meta tag. A crawler has to fetch the page to see that tag.
  • It does not protect anything. It is a public request that well-behaved crawlers follow. It also advertises the paths you list. Protect private content with authentication.
  • A single stray line can hide a whole site. Disallow: / under User-agent: * blocks all compliant crawlers. It is common on staging sites and occasionally survives into production.

Sitemaps help discovery

An XML sitemap lists the URLs you want search engines to know about. It matters most for new sites with few inbound links, large sites, and pages that are poorly linked internally. The sitemap generator produces a standard file from a list of URLs, with optional last-modified dates.

  • List only canonical URLs that return status 200 and that you want indexed. Leave out redirects, error pages, noindex pages, and parameter variants.
  • Use lastmod honestly. Search engines learn to ignore dates that change on every deployment.
  • Google ignores priority and changefreq, so there is no need to tune them.
  • Reference the sitemap in robots.txt and submit it in Google Search Console and Bing Webmaster Tools, which also report indexing problems.

A sitemap is a hint, not a command. Listing a page does not guarantee that it will be indexed. Quality, usefulness, and internal links decide that.

Meta tags describe the page

In the page’s <head>, a few tags do most of the work. The meta tag generator writes them with correct escaping:

  • <title>: the most important on-page signal and usually the clickable headline in results. Make it unique per page, lead with the distinctive words, and aim for roughly 50–60 characters.
  • <meta name="description">: often used as the snippet under the title. It does not affect ranking directly, but a clear description improves the click-through rate. Aim for around 120–160 characters.
  • <link rel="canonical">: names the preferred URL when the same content is reachable at several addresses, such as with tracking parameters, trailing slashes, or http and https.
  • <meta name="robots">: noindex keeps a page out of results, and nofollow asks crawlers not to follow its links.

Search engines rewrite titles and descriptions when they judge another text more relevant to a query, so treat the preview as a goal rather than a guarantee. The keywords meta tag has been ignored by Google for well over a decade.

Social sharing tags

When a link is shared in Slack, LinkedIn, WhatsApp, or Facebook, those apps read Open Graph tags (og:title, og:description, og:image) to build a preview card. They do not affect search rankings, but a good card makes a shared link far more likely to be clicked. The social card preview writes the tags and shows an approximate card. A 1200 × 630 pixel image works well across platforms.

A launch checklist

  1. robots.txt exists and does not block the site or its CSS and JavaScript.
  2. The sitemap lists every page you want indexed, and only those.
  3. Every page has a unique title, a description, and a canonical URL.
  4. Pages you want hidden use noindex rather than a robots.txt block.
  5. Open Graph tags and an image are set for pages people will share.
  6. The site is verified in Google Search Console and the sitemap is submitted, so you can watch indexing and fix errors.