What is a robots.txt file?
It is a text file at https://example.com/robots.txt with rules for crawlers.
User-agent: *
Disallow: /admin/
Sitemap: https://example.com/sitemap.xml
Test result of the rules above
Googlebot /admin/settings Blocked
Googlebot /blog/ Allowed
Blocking a page in robots.txt does not hide it from search, and it does not protect private data. See Robots.txt.
Learn more about the similar topics:
| Tutorials |
|---|
| No Content Found. |
| Exercises & Assignments |
|---|
| No Content Found. |
| Interview Questions & Answers |
|---|
| No Content Found. |