I have a new Wordpress 3.5.1 install (hosted on Dreamhost FWIW) that I do not want to be indexed by search engines. I would like to serve a simple robots.txt with
Disallow: / for all user agents.
I have checked the "Discourage search engines from indexing this site" box on the Settings > Reading menu, but http://mysite.com/robots.txt still returns a 404.
Is there a way to have Wordpress automatically generate and serve an appropriate robots.txt file? If not, what is the best way to configure it to serve my own static robots.txt file?