SEO – Robots.txt
The robots.txt file is a small text file in the root of your website. It tells crawlers which parts of the site they may visit. It is a request, not a lock. Good bots like Googlebot follow it, but bad bots can ignore it.
The address is always at the root: https://example.com/robots.txt. A file in a sub folder is not used.
Syntax
| Rule | Meaning |
|---|---|
User-agent: | Which bot the rules are for. * means all bots |
Disallow: | A path the bot should not crawl |
Allow: | A path the bot may crawl (useful to make an exception inside a blocked folder) |
Sitemap: | The full address of your XML sitemap |
# | A comment |
Google supports only these four fields. Other lines, like Crawl-delay or noindex, are ignored by Google.
Example 1: allow everything
User-agent: *
Disallow:
An empty Disallow means nothing is blocked.
Example 2: block the whole site
User-agent: *
Disallow: /
This is useful for a test site, but be careful. Many sites were removed from search because this file was copied to the live site by mistake.
Example 3: block some folders and allow one file
User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /search
Allow: /admin/help.html
Sitemap: https://example.com/sitemap.xml
Here is a test of the rules above. We test some paths for Googlebot. When rules conflict, Google uses the most specific one (the longest matching path), and Allow wins if both are equal.
Googlebot /index.html Allowed
Googlebot /admin/login.php Blocked
Googlebot /admin/help.html Allowed
Googlebot /cart/checkout Blocked
Googlebot /search?q=seo Blocked
Googlebot /blog/seo-tips/ Allowed
Example 4: different rules for different bots
User-agent: Googlebot
Disallow: /private/
User-agent: *
Disallow: /private/
Disallow: /drafts/
Googlebot /drafts/post-1 Allowed
Googlebot /private/file.pdf Blocked
Otherbot /drafts/post-1 Blocked
Otherbot /blog/ Allowed
A bot follows only the group which matches its own name best. Here Googlebot uses the first group only, so /drafts/ is open for it.
Wildcards
*matches any text.Disallow: /*?sort=blocks all URLs which have?sort=.$means the end of the URL.Disallow: /*.pdf$blocks all PDF files.
User-agent: *
Disallow: /*.pdf$
Disallow: /*?sort=
Googlebot /files/guide.pdf Blocked
Googlebot /files/guide.pdf.html Allowed
Googlebot /shop/?sort=price Blocked
Googlebot /shop/ Allowed
Important points
- Robots.txt does not hide pages. A blocked page can still appear in the results as a bare link, if other sites link to it. To keep a page out of search, use a
noindexmeta tag, and allow the crawl so Google can see the tag. - It is not for security. Anyone can read your robots.txt. Do not list secret folders in it. Protect them with a password.
- Do not block CSS and JavaScript files which are needed to show the page. Google needs them to understand the page.
- Paths are case sensitive:
/Admin/and/admin/are different. - The file must be plain text, in UTF-8, and return a normal 200 status.
Bots for AI companies
Some AI companies publish the name of their crawlers, so you can allow or block them in robots.txt, for example GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl) and Google-Extended (a control for Google’s AI training). Check the website of each company for current names, because they change. Not every bot follows the rules.
User-agent: GPTBot
Disallow: /
User-agent: *
Disallow:
Test your robots.txt
- Open
yourdomain.com/robots.txtin a browser, and read it. - In Google Search Console, the robots.txt report shows if Google could read the file and which version it has.
- The URL Inspection tool tells if a page is blocked.