SEOMedium

robots.txt missing or misconfigured on WordPress

The robots.txt file is absent, inaccessible or does not contain a Sitemap directive - limiting the control you have over your site's indexing.

Why it matters

Without robots.txt, Google potentially indexes useless pages (/wp-admin, /wp-login, search result pages) and does not easily find your sitemap.

How to fix

  1. 1

    Recommended robots.txt for WordPress

    text
    User-agent: *
Disallow: /wp-admin/
Disallow: /wp-login.php
Disallow: /xmlrpc.php
Disallow: /?s=
Disallow: /search/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://yoursite.com/sitemap.xml
  2. 2

    Create/edit via Yoast SEO

    Yoast SEO → Tools → File editor → "robots.txt" tab. More convenient than editing the file directly.

  3. 3

    Create the file manually

    Create robots.txt at the root of your site via FTP or SSH. Verify it is accessible at yoursite.com/robots.txt.

Related articles

SEOXML sitemap missing or unreferencedSEOMissing canonical URL on WordPress

Ready to fix this issue on your site?

Audit my site for free →