Home » FAQs » SEO Interview Questions » All SEO Interview Questions and Answers » What is a robots.txt file?

What is a robots.txt file?

It is a text file at https://example.com/robots.txt with rules for crawlers.

User-agent: *
Disallow: /admin/
Sitemap: https://example.com/sitemap.xml
Test result of the rules above

Googlebot /admin/settings Blocked
Googlebot /blog/ Allowed

Blocking a page in robots.txt does not hide it from search, and it does not protect private data. See Robots.txt.

Share this post:

Share on LinkedIn
Learn more about the similar topics:
Tutorials
No Content Found.
Exercises & Assignments
No Content Found.
Interview Questions & Answers
No Content Found.