Home » Codes » SEO » SEO Top Codes » Robots.txt Template for WordPress

Robots.txt Template for WordPress

This robots.txt keeps crawlers out of the admin area and the internal search pages, but allows the one admin file that many themes and plugins need on the front end.

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /?s=
Disallow: /search/

Sitemap: https://example.com/wp-sitemap.xml

Here is the test of some paths for Googlebot:

Test result of the rules above

Googlebot /wp-admin/options.php Blocked
Googlebot /wp-admin/admin-ajax.php Allowed
Googlebot /?s=seo Blocked
Googlebot /search/seo/ Blocked
Googlebot /wp-content/uploads/2026/05/photo.jpg Allowed
Googlebot /blog/seo-guide/ Allowed

How it works

  • Disallow: /wp-admin/ blocks the admin folder, and the longer Allow rule makes an exception for admin-ajax.php.
  • /?s= and /search/ are the internal search result pages. They give no value in Google and waste crawling.
  • The images in /wp-content/uploads/ stay open, so Google can use them in image search.
  • Change the sitemap address to your own. If you use Yoast SEO or Rank Math, it is usually /sitemap_index.xml.
Note: Do not block /wp-content/ or /wp-includes/. Google needs the CSS and JavaScript from there to render your pages. See Robots.txt.

Share this post:

Share on LinkedIn
Learn more about the similar topics:
Tutorials
No Content Found.
Exercises & Assignments
No Content Found.
Interview Questions & Answers
No Content Found.