SEO – Robots.txt

The robots.txt file is a small text file in the root of your website. It tells crawlers which parts of the site they may visit. It is a request, not a lock. Good bots like Googlebot follow it, but bad bots can ignore it.

The address is always at the root: https://example.com/robots.txt. A file in a sub folder is not used.

Syntax

RuleMeaning
User-agent:Which bot the rules are for. * means all bots
Disallow:A path the bot should not crawl
Allow:A path the bot may crawl (useful to make an exception inside a blocked folder)
Sitemap:The full address of your XML sitemap
#A comment

Google supports only these four fields. Other lines, like Crawl-delay or noindex, are ignored by Google.

Example 1: allow everything

User-agent: *
Disallow:

An empty Disallow means nothing is blocked.

Example 2: block the whole site

User-agent: *
Disallow: /

This is useful for a test site, but be careful. Many sites were removed from search because this file was copied to the live site by mistake.

Example 3: block some folders and allow one file

User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /search
Allow: /admin/help.html

Sitemap: https://example.com/sitemap.xml

Here is a test of the rules above. We test some paths for Googlebot. When rules conflict, Google uses the most specific one (the longest matching path), and Allow wins if both are equal.

Test result of the rules above

Googlebot /index.html Allowed
Googlebot /admin/login.php Blocked
Googlebot /admin/help.html Allowed
Googlebot /cart/checkout Blocked
Googlebot /search?q=seo Blocked
Googlebot /blog/seo-tips/ Allowed

Example 4: different rules for different bots

User-agent: Googlebot
Disallow: /private/

User-agent: *
Disallow: /private/
Disallow: /drafts/
Test result of the rules above

Googlebot /drafts/post-1 Allowed
Googlebot /private/file.pdf Blocked
Otherbot /drafts/post-1 Blocked
Otherbot /blog/ Allowed

A bot follows only the group which matches its own name best. Here Googlebot uses the first group only, so /drafts/ is open for it.

Wildcards

  • * matches any text. Disallow: /*?sort= blocks all URLs which have ?sort=.
  • $ means the end of the URL. Disallow: /*.pdf$ blocks all PDF files.
User-agent: *
Disallow: /*.pdf$
Disallow: /*?sort=
Test result of the rules above

Googlebot /files/guide.pdf Blocked
Googlebot /files/guide.pdf.html Allowed
Googlebot /shop/?sort=price Blocked
Googlebot /shop/ Allowed

Important points

  • Robots.txt does not hide pages. A blocked page can still appear in the results as a bare link, if other sites link to it. To keep a page out of search, use a noindex meta tag, and allow the crawl so Google can see the tag.
  • It is not for security. Anyone can read your robots.txt. Do not list secret folders in it. Protect them with a password.
  • Do not block CSS and JavaScript files which are needed to show the page. Google needs them to understand the page.
  • Paths are case sensitive: /Admin/ and /admin/ are different.
  • The file must be plain text, in UTF-8, and return a normal 200 status.

Bots for AI companies

Some AI companies publish the name of their crawlers, so you can allow or block them in robots.txt, for example GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl) and Google-Extended (a control for Google’s AI training). Check the website of each company for current names, because they change. Not every bot follows the rules.

User-agent: GPTBot
Disallow: /

User-agent: *
Disallow:

Test your robots.txt

  • Open yourdomain.com/robots.txt in a browser, and read it.
  • In Google Search Console, the robots.txt report shows if Google could read the file and which version it has.
  • The URL Inspection tool tells if a page is blocked.
Note: For a WordPress site, the default robots.txt is small. You usually do not need to block much. Block only what really wastes the crawl, like internal search pages or cart pages, and add the sitemap line. Next: XML Sitemap.

Congratulations! Chapter Finished. Learn more about the similar topics:
Exercises & Assignments
Write a robots.txt file to block folders
Block all PDF files and URLs with a parameter in robots.txt
Add a canonical tag to a filtered page
Create an XML sitemap for three pages
Interview Questions & Answers
Which is more useful between an HTML sitemap and an XML sitemap?
What is technical SEO?
What is a robots.txt file?
What is an XML sitemap?
What is a canonical tag?
What is crawl budget?
What is duplicate content and how do you fix it?