0

This is happening with a fresh, new WordPress install. But all you need to do is look at the crazy do_robots() function, which outputs the following...

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://knowingart.com/wp-sitemap.xml

Sending robots to "wp-admin" (by default) makes no sense:

  1. Why would I send random robots into my admin directory?! The admin directory is for me, and me alone. Robots/crawlers should not be in my admin directory doing anything.
  2. Why would I DOS attack myself by sending robots to an admin script?
  3. If I was a bot developer, I might (wrongly) interpret this "allow" as negation of the disallow, because the allow comes after the disallow and it's the same directory. Why does robots.txt contradict itself here?
  4. This weird (default) robots.txt seems to break DuckDuckGo. For example the top search result is my wp-admin directory?! It appears DuckDuckGo read my robots.txt, went into wp-admin because robots.txt told it to go there, and that is the wrong directory. Was DDG's crawler confused by the weird robots.txt file? Now I'm thinking DDG crawled my blog before any content was available, and just hasn't updated yet, that seems to be a more likely explanation.

Why does WordPress send robot crawlers to an admin directory that has no content?! It makes no sense to me, which is why I am here trying to figure it out. I can only imagine the author of this do_robots() code doesn't understand the purpose of robots.txt

5
  • 1
    core.trac.wordpress.org/ticket/18465 and core.trac.wordpress.org/ticket/33156 seems to give some insights
    – birgire
    Mar 15 at 22:02
  • 1
    upvoting the question to get it to a zero score, but next time you should probably drink some water first to make sure you relaxed enough and your question will not end sounding like a rant ;). Mar 16 at 3:33
  • @MarkKaplun Thanks, I take my job seriously. Since I started investigating this, it seems Google is partly responsible for adding this new "sitemap:" line, because I think they coded the new wp-sitemap.xml, which is kind of a big deal in my opinion. That new sitemap code is impressive (though a little buggy) I had no idea it was there. That said, I am currently taking some medication for a minor health issue, yes I was cranky and went back to tone it down a little. However, I am justified as a WordPress hosting system admin and web developer to care about WordPress config, SEO, sitemaps, etc.
    – PJ Brunet
    Mar 16 at 19:45
  • @birgire I think that last ticket is the best answer. It shows this decision was made without much thought, and other people questioned the wisdom of this decision (not just me) and this admin-ajax thing was a knee-jerk reaction to some Google Webmaster Tools email blast that went out by mistake, due to themes linking to admin files. In my opinion, robots.txt should be the responsibility of the system admin. Maybe I can "touch robots.txt" but will that break wp-sitemap.xml? LOL
    – PJ Brunet
    Mar 16 at 20:15
  • @PJBrunet all kinds of plugins can edit he file (it is not a file actually, it is auto generated at each request to the URL). A good plugin will not change the link to the sitemap. Not recomending a plugin because I write my own code to modify things when needed so have no idea which would be a good plugin Mar 17 at 3:19

2 Answers 2

3

the truth is that probably nothing should be blocked in robots.txt by core (this is IIRC was joost's position abot the matter) as wordpress is an open platform and front end content and styling might and is generated in all kinds of directories which might not make much sense to you and me. Wordpress is not in the buisness of preventing site owners from installing badly written plugins.

Why do you have pages indexed by a search engine? Wordpress uses a kind of "don't index" headers for all admin pages so most likely you have some badly written code that prevents the header from bein sent. (this assumes that there is no bug in bing which is the SE which powers DDG).

Maybe worth reminding that robots.txt is just an advisory file, it is up to the search engine to decide if and how to respect it. IIRC google will not respect it fully if there was a link to a page which supposed to be excluded by robots.txt

3
  • I agree "nothing should be blocked" in fact WordPress should not auto-generate robots.txt at all. The answer is: WordPress made a mistake. Historically, the purpose of robots.txt is to advise crawlers. Yet I hear a lot of people talking about robots.txt in relation to themes, the "front end" etc. Even if there was a problem with Google nagging about something by email via Webmaster Tools, that's not the role of robots.txt (Googlebot is just one crawler among many.) And of course if there are Googlebot issues that can't be resolved with Google, NginX/Apache/etc can mange those things.
    – PJ Brunet
    Mar 16 at 20:01
  • "Most likely you have some badly written code" No, it's fresh WordPress install from latest.tar.gz, using a theme by Automattic downloaded via the dashboard. That said, looks like DDG already updated itself and corrected the error.
    – PJ Brunet
    Mar 16 at 20:21
  • @PJBrunet if its a fresh install I have no idea what might be going on. For sure it do not supposed to happen but if for whatever reason yo have incoming links to those urls SE are more likely to do whatever they want. Mar 17 at 3:14
2

First: have a read-up on robots.txt, if you haven't. Another good reference is this one from Yoast.

The sample you posted:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://knowingart.com/wp-sitemap.xml

...tells all user-agents (ie, all crawlers) that they are not allowed to crawl anything in your wp-admin directory except /wp-admin/admin-ajax.php, which is required by WordPress for properly-functioning AJAX.

It's not telling robots to go to your /wp-admin/ directory; it's telling them they're unwelcome there (except for the AJAX requirement).

As for your question #4, I'm afraid I don't have an answer to that one.

5
  • 2
    @PJBrunet admin-ajax.php is very commonly used for public AJAX requests on the front end of the website. If requests to this file from bots are disallowed, then when a search engine bot crawls your site these requests would not work and the bot would potentially be looking at a broken version of the page. Mar 15 at 22:38
  • robots.txt has nothing to do with WordPress accessing itself. It's meant as instructions for search engine crawlers, letting them know where they can (and, more importantly) cannot go on a site.
    – Pat J
    Mar 15 at 22:55
  • @JacobPeattie I see Jacob's point, Google is doing some special magic to render the page and somehow analyze the UI, which has been true since Matt Cutts days. However, the role of robots.txt is not to please Googlebot specifically. If that is really an issue, let the theme developer fix it, or take up the issue with Google. I realize Google SEO is very important to people, but it's just one robot among many--and frankly I don't care if somebody's theme needs admin-ajax.php to render correctly. Theme-specific SEO problems (themes that don't fail gracefully) should not creep into robots.txt
    – PJ Brunet
    Mar 16 at 20:51
  • While I upvoted your answer, I see problems here. First, my concern about the order of the allow/disallow is a real issue core.trac.wordpress.org/ticket/33156#comment:18 Regardless of the robots.txt specification, it's best to be specific and clear because every robot will interpret robots.txt in its own way, regardless of the spec.
    – PJ Brunet
    Mar 16 at 21:09
  • As for the AJAX issue, in relation to themes rendering correctly for Googlebot or because of a glitch in Webmaster Tools, neither problem belongs in robots.txt. AJAX pages do not need AJAX to render for a search engine. "Lazy loading" content should not break Googlebot. AJAX devs could and should load placeholder content, in other words AJAX content sections can and should fail gracefully. Badly coded themes should not pollute robots.txt
    – PJ Brunet
    Mar 16 at 21:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.