0

I created a page on a Wordpress site that was for internal use only and triggers some backend code. Within a few days I started seeing hits on that page from "bingbot".

I'm not using any kind of sitemap plugin. How are crawlers finding this page?

I know the robots.txt file can block them but I want to make sure they don't show up for crawlers that don't respect this. I still want to have the page publicly accessible if someone types in the URL.

What needs to be done in Wordpress to make sure a page can't be discovered except by typing in the URL?

  • " to make sure a page can't be discovered except by typing in the URL?" - but that is exactly what many crawlers are doing. The way to go is usually either a robots.txt or some restrictions like HTTP Basic Auth (can be a simple user/password) – kero Jun 18 at 6:21
  • Crawlers can't guess the URL, they have to get it from somewhere. That is what I would like to remove. – user83358 Jun 18 at 22:33
  • This is just not correct. I have had several websites in development that suddenly appeared in Google. Even if they do not guess URLs (which I presume they do), if you surf the site in Chrome, they know it exists. The only effective way of blocking them out is by blocking out all random visitors (eg via Basic Auth) – kero Jun 19 at 6:45
  • If you create the url yoursite.com/dfoimsfg9q743jroaiemf9w84mr09wqnmr09 will it show up in a crawler? Why do you think that is? The URL that I created was only for an internal function, can't use basic auth, and was only accessed through Safari for testing. Since Apple doesn't have a search engine that seems like an unlikely way for it to be discovered. – user83358 Jun 20 at 15:42
  • After creating a page with a completely random URL, and not accessing it at all, it took 4 days to start getting hits from bing bot. Since the server hasn't been hammered with trillions of requests for random URLs, this can only mean that crawlers (and potentially malicious attackers) are getting a list of all pages that exist in Wordpress, whether or not they are linked from anywhere else on the public site. That could be a concern to some people. – user83358 Jun 28 at 0:51
-1

Yes, if they are public pages WordPress will let them be indexed unless you tell it not to.

Try adding this to you function.php this:

function allow_no_robots() {

    global $post;

    if ( $post->ID == PAGE_ID_TO_NOT_INDEX ) {
        echo '<meta name="robots" content="noindex, nofollow">';

} add_action( 'wp_head', 'allow_no_robots' );

That sould do it.

  • That seems like it would take effect once a crawler accesses the page. I want to stop the crawler from finding the page URL. – user83358 Jun 18 at 22:34
  • Wordpress has built-in function for this. Please read Codex – spoilt Jun 19 at 12:48
  • is that a built in function to add the meta tag? – user83358 Jun 20 at 15:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.