Home » Exercises » SEO Exercises » All SEO Exercises » Write a robots.txt file to block folders

Write a robots.txt file to block folders

Description: Write a robots.txt for all bots which blocks the folders /admin/ and /tmp/, but allows the file /admin/help.html. Add the sitemap https://example.com/sitemap.xml.

Hint:

  • Use Disallow for blocked folders, Allow for the exception and Sitemap with the full address.

Solution

User-agent: *
Disallow: /admin/
Disallow: /tmp/
Allow: /admin/help.html

Sitemap: https://example.com/sitemap.xml
Test result of the rules above

Googlebot /admin/users Blocked
Googlebot /admin/help.html Allowed
Googlebot /tmp/cache.txt Blocked
Googlebot /about/ Allowed

How it works

  • The longer rule Allow: /admin/help.html wins over the shorter Disallow: /admin/ for that file.
  • The test shows that only the intended paths are blocked.

Read Robots.txt.

Share this post:

Share on LinkedIn
Learn more about the similar topics:
Tutorials
No Content Found.
Exercises & Assignments
No Content Found.
Interview Questions & Answers
No Content Found.