This is a file that contains directions for search spiders. Its content depends on whether the user has allowed search engines to crawl the site.
This is a file that contains directions for search spiders. Its content depends on whether the user has allowed search engines to crawl the site, either through the WP interface or a plugin.
Note that robots.txt does not actively block the site. Search engines keep to its rules out of their own design.
