0

My website is setup where wordpress is installed in it's own subdirectory, but the content is served as though it's at the domain root.
(This wordpress codex page details the configuration)

Wordpress Url: https://www.example.com/wordpress
Site Url: https://www.example.com

So in this case, urls get re-written to NOT include /wordpress in the url it serves pages from (although image urls do include it: https://www.example.com/wordpress/wp-content/uploads/...).

URL for a page is: www.example.com/mypage/
NOT: www.example.com/wordpress/mypage/
but image link is: www.example.com/wordpress/wp-content/uploads/2018/12/25/image1.jpg

Question:
Do I put robots.txt and sitemap.xml in the "webiste root" directory on my server (example.com)
or its wordpress directory (example.com/wordpress) ?

I cannot find this particular case in any searches.
Also trial and error is not helping, because I get conflicting results from various sitemap and robot validators.

Additionally, it seems some plugin or some part of WordPress insists on creating Virtual sitemap (sitemap.xml.gz) and robot (robot.txt) files, which further exasperates my efforts to definitively determine the proper location to place these files for this common, but non-standard of setup.

Improve this question

1 Answer 1

Reset to default
1

You can safely place your robot.txt file as well as the sitemap.xml file into your websites root directory and NOT the WordPress directory.

When a robot visits your site, the first place it checks is http://sitename.com/robots.txt, hence you want to keep these files at the root of your domain.

You can also disallow what the robot has access to 

    User-agent: *    # applies to all robots
    Disallow: /      # disallow indexing of all pages

    Disallow: wordpress/wp-admin/
    Disallow: wordpress/wp-includes/
    Disallow: wordpress/wp-content/plugins/

W3.org

Improve this answer
13
  • Hmm... The logic of your explanation sounds inverted to me. Could you help me understand where/how I'm thinking about this incorrectly? The way I see it, htaccess intercepts the request for /sitemap.xml but instead (as usual) serves up files from /wordpress, ie: /wordpress/sitemap.xml, Which implies in my mind that I would need to place sitemap and robot files inside /wordpress. Not root.
    – SherylHohman
    Commented Dec 9, 2018 at 2:12
  • The way I understand it, I actually never actually serve anything from root. (htaccess and index.php prevent that?). Then again, the only stuff I (usually) ever (intentionally) serve are from WordPress. So I don't really know how "it" works for any non-wordpress-generated files/requests. Thanks for helping me clear my foggy thinking.
    – SherylHohman
    Commented Dec 9, 2018 at 2:12
  • So wait, you're saying my server finds the file in root, then passes the file itself to WordPress ? Then WordPress handles it?
    – SherylHohman
    Commented Dec 9, 2018 at 2:16
  • Ok. so more experimentation. It looks like if a requested file exists in the root directory, it just serves it up. If not, it forwards the request to /wordpress where WordPress intercepts the request, and attempts to process it. If it is an existing page, it serves that page, otherwise it serves a 404 page (usually styled by the theme). I got at this by placing file1.txt in root, and file2.txt in /wordpress. file1 displays its contents in my browser;but file2.txt showed me a 404 page from my site.
    – SherylHohman
    Commented Dec 9, 2018 at 2:44
  • So, if I have sitemap.xml in root, my server probably just serves that right up. Same for robots.txt. If I don't have sitemap.xml in root, WordPress might generate one (depending on plugins or maybe even wordpress itself). And either way, I'm guessing any sitemap.xml file placed in /wordpress is probably never acknowledged.
    – SherylHohman
    Commented Dec 9, 2018 at 2:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.