robots.txt is a plain text file at the root of a site — example.com/robots.txt — that tells crawlers which parts of the site they should not request. It is the first thing most crawlers look for when they arrive.
# Applies to all crawlers
User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /search?
# Where the sitemap lives
Sitemap: https://example.com/sitemap.xmlUser-agent — which crawler the following rules apply to. * means all of them.Disallow — a path prefix crawlers should not request.Allow — an exception carved out of a broader Disallow.Sitemap — where your sitemap is. Covered in the next lesson.The genuine uses are narrower than people assume:
The most misunderstood point in SEO
robots.txt does not make a page private and does not reliably keep it out of search results. It asks well-behaved crawlers not to fetch a URL. A blocked page can still appear in results if other sites link to it, because the engine knows the address exists even though it has not read the content. To keep a page out of the index, use a noindex meta tag — and note that a page blocked in robots.txt can never be seen to have that tag, since the crawler is not allowed to read it.
User-agent: *
Disallow: /Those two lines ask every crawler to stay away from the entire site. This is a completely legitimate configuration for a staging or development site — and it is disastrous when it reaches production, which happens more often than it should, usually because a staging file was copied across during a launch.
Checking yoursite.com/robots.txt immediately after any launch or migration takes ten seconds and catches this before it costs months.
robots.txt is publicly readable by anyone. Listing Disallow: /secret-admin-panel/ in it announces the existence of that path to the whole world. Anything genuinely sensitive needs authentication, not a crawler directive.