This is happening with a fresh, new WordPress install. But all you need to do is look at the crazy do_robots() function, which outputs the following...
User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Sitemap: https://knowingart.com/wp-sitemap.xml
Sending robots to "wp-admin" (by default) makes no sense:
- Why would I send random robots into my admin directory?! The admin directory is for me, and me alone. Robots/crawlers should not be in my admin directory doing anything.
- Why would I DOS attack myself by sending robots to an admin script?
- If I was a bot developer, I might (wrongly) interpret this "allow" as negation of the disallow, because the allow comes after the disallow and it's the same directory. Why does robots.txt contradict itself here?
- This weird (default) robots.txt seems to break DuckDuckGo. For example the top search result is my wp-admin directory?! It appears DuckDuckGo read my robots.txt, went into wp-admin because robots.txt told it to go there, and that is the wrong directory. Was DDG's crawler confused by the weird robots.txt file? Now I'm thinking DDG crawled my blog before any content was available, and just hasn't updated yet, that seems to be a more likely explanation.
Why does WordPress send robot crawlers to an admin directory that has no content?! It makes no sense to me, which is why I am here trying to figure it out. I can only imagine the author of this do_robots() code doesn't understand the purpose of robots.txt