on every WordPress installation there is a default virtual
robots.txt file with the following
User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php
but on my 2 last project
/robots.txt give me a 404 instead.
- I tried to disable all plugin
- I tried to switch theme with a blank one
- Tried to look on another older project which I updated last week and robots.txt is working
This not server related because it work on my local environment with older project
Any idea why ?