7

This is happening with a fresh, new WordPress install. But all you need to do is look at the crazy do_robots() function, which outputs the following...

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://knowingart.com/wp-sitemap.xml

Sending robots to "wp-admin" (by default) makes no sense:

  1. Why would I send random robots into my admin directory?! The admin directory is for me, and me alone. Robots/crawlers should not be in my admin directory doing anything.
  2. Why would I DOS attack myself by sending robots to an admin script?
  3. If I was a bot developer, I might (wrongly) interpret this "allow" as negation of the disallow, because the allow comes after the disallow and it's the same directory. Why does robots.txt contradict itself here?
  4. This weird (default) robots.txt seems to break DuckDuckGo. For example the top search result is my wp-admin directory?! It appears DuckDuckGo read my robots.txt, went into wp-admin because robots.txt told it to go there, and that is the wrong directory. Was DDG's crawler confused by the weird robots.txt file? Now I'm thinking DDG crawled my blog before any content was available, and just hasn't updated yet, that seems to be a more likely explanation.

Why does WordPress send robot crawlers to an admin directory that has no content?! It makes no sense to me, which is why I am here trying to figure it out. I can only imagine the author of this do_robots() code doesn't understand the purpose of robots.txt

Improve this question
6
  • 2
    core.trac.wordpress.org/ticket/18465 and core.trac.wordpress.org/ticket/33156 seems to give some insights
    – birgire
    Commented Mar 15, 2022 at 22:02
  • 3
    upvoting the question to get it to a zero score, but next time you should probably drink some water first to make sure you relaxed enough and your question will not end sounding like a rant ;).
    – Mark Kaplun
    Commented Mar 16, 2022 at 3:33
  • @MarkKaplun Thanks, I take my job seriously. Since I started investigating this, it seems Google is partly responsible for adding this new "sitemap:" line, because I think they coded the new wp-sitemap.xml, which is kind of a big deal in my opinion. That new sitemap code is impressive (though a little buggy) I had no idea it was there. That said, I am currently taking some medication for a minor health issue, yes I was cranky and went back to tone it down a little. However, I am justified as a WordPress hosting system admin and web developer to care about WordPress config, SEO, sitemaps, etc.
    – Jay Brunet
    Commented Mar 16, 2022 at 19:45
  • 1
    @birgire I think that last ticket is the best answer. It shows this decision was made without much thought, and other people questioned the wisdom of this decision (not just me) and this admin-ajax thing was a knee-jerk reaction to some Google Webmaster Tools email blast that went out by mistake, due to themes linking to admin files. In my opinion, robots.txt should be the responsibility of the system admin. Maybe I can "touch robots.txt" but will that break wp-sitemap.xml? LOL
    – Jay Brunet
    Commented Mar 16, 2022 at 20:15
  • @PJBrunet all kinds of plugins can edit he file (it is not a file actually, it is auto generated at each request to the URL). A good plugin will not change the link to the sitemap. Not recomending a plugin because I write my own code to modify things when needed so have no idea which would be a good plugin
    – Mark Kaplun
    Commented Mar 17, 2022 at 3:19

3 Answers 3

Reset to default
7

First: have a read-up on robots.txt, if you haven't. Another good reference is this one from Yoast.

The sample you posted:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://knowingart.com/wp-sitemap.xml

...tells all user-agents (ie, all crawlers) that they are not allowed to crawl anything in your wp-admin directory except /wp-admin/admin-ajax.php, which is required by WordPress for properly-functioning AJAX.

It's not telling robots to go to your /wp-admin/ directory; it's telling them they're unwelcome there (except for the AJAX requirement).

As for your question #4, I'm afraid I don't have an answer to that one.

Improve this answer
5
  • 2
    @PJBrunet admin-ajax.php is very commonly used for public AJAX requests on the front end of the website. If requests to this file from bots are disallowed, then when a search engine bot crawls your site these requests would not work and the bot would potentially be looking at a broken version of the page.
    – Jacob Peattie
    Commented Mar 15, 2022 at 22:38
  • robots.txt has nothing to do with WordPress accessing itself. It's meant as instructions for search engine crawlers, letting them know where they can (and, more importantly) cannot go on a site.
    – Pat J
    Commented Mar 15, 2022 at 22:55
  • @JacobPeattie I see Jacob's point, Google is doing some special magic to render the page and somehow analyze the UI, which has been true since Matt Cutts days. However, the role of robots.txt is not to please Googlebot specifically. If that is really an issue, let the theme developer fix it, or take up the issue with Google. I realize Google SEO is very important to people, but it's just one robot among many--and frankly I don't care if somebody's theme needs admin-ajax.php to render correctly. Theme-specific SEO problems (themes that don't fail gracefully) should not creep into robots.txt
    – Jay Brunet
    Commented Mar 16, 2022 at 20:51
  • While I upvoted your answer, I see problems here. First, my concern about the order of the allow/disallow is a real issue core.trac.wordpress.org/ticket/33156#comment:18 Regardless of the robots.txt specification, it's best to be specific and clear because every robot will interpret robots.txt in its own way, regardless of the spec.
    – Jay Brunet
    Commented Mar 16, 2022 at 21:09
  • As for the AJAX issue, in relation to themes rendering correctly for Googlebot or because of a glitch in Webmaster Tools, neither problem belongs in robots.txt. AJAX pages do not need AJAX to render for a search engine. "Lazy loading" content should not break Googlebot. AJAX devs could and should load placeholder content, in other words AJAX content sections can and should fail gracefully. Badly coded themes should not pollute robots.txt
    – Jay Brunet
    Commented Mar 16, 2022 at 21:14
3

the truth is that probably nothing should be blocked in robots.txt by core (this is IIRC was joost's position abot the matter) as wordpress is an open platform and front end content and styling might and is generated in all kinds of directories which might not make much sense to you and me. Wordpress is not in the buisness of preventing site owners from installing badly written plugins.

Why do you have pages indexed by a search engine? Wordpress uses a kind of "don't index" headers for all admin pages so most likely you have some badly written code that prevents the header from bein sent. (this assumes that there is no bug in bing which is the SE which powers DDG).

Maybe worth reminding that robots.txt is just an advisory file, it is up to the search engine to decide if and how to respect it. IIRC google will not respect it fully if there was a link to a page which supposed to be excluded by robots.txt

Improve this answer
3
  • I agree "nothing should be blocked" in fact WordPress should not auto-generate robots.txt at all. The answer is: WordPress made a mistake. Historically, the purpose of robots.txt is to advise crawlers. Yet I hear a lot of people talking about robots.txt in relation to themes, the "front end" etc. Even if there was a problem with Google nagging about something by email via Webmaster Tools, that's not the role of robots.txt (Googlebot is just one crawler among many.) And of course if there are Googlebot issues that can't be resolved with Google, NginX/Apache/etc can mange those things.
    – Jay Brunet
    Commented Mar 16, 2022 at 20:01
  • "Most likely you have some badly written code" No, it's fresh WordPress install from latest.tar.gz, using a theme by Automattic downloaded via the dashboard. That said, looks like DDG already updated itself and corrected the error.
    – Jay Brunet
    Commented Mar 16, 2022 at 20:21
  • @PJBrunet if its a fresh install I have no idea what might be going on. For sure it do not supposed to happen but if for whatever reason yo have incoming links to those urls SE are more likely to do whatever they want.
    – Mark Kaplun
    Commented Mar 17, 2022 at 3:14
0

You should allow admin-ajax.php to speak with the search engine in WordPress. Actually it's a request that is working with themes / plugins and search engines if you update the WordPress website.

Improve this answer
1
  • 1
    OK, I can see it's needed for the site to work but I can't imagine you actually get useful indexable content out of admin-ajax.
    – Rup
    Commented Jan 22 at 10:27

Not the answer you're looking for? Browse other questions tagged or ask your own question.