0

We have a site on wpegnine that gets about 20k visits a day and heavily depends on ajax for our inbound marketing strategy (lead/visitor processing).

These ajax calls are killing our server performance.

I believe that a good way to alleviate our problem is to disable all AJAX calls to wp-admin for spiders.

They don't need to see our dynamic calls to action and we don't need any visitor tracking on these spiders.

How do we setup our robots.txt, or even better, code in a solution that will keep spiders from using these resources?

  • 1
    1. send ajax requests with POST method 2. use ajax api and use robots.txt to disallow requests to /wp-admin/*: using that API the url is like /wp-admin/admin-ajax.php 3. use nonce checking for ajax requests: search engines can't predict nonces. If you want decrease nonce life using 'nonce_life' filter. All of these are best practices and prevent search engines from running ajax is a welcome side effect. – gmazzap Mar 23 '14 at 21:46
  • We do use POST methods and the WordPress AJAX API. But how does using POST method help alone? Does it have to be used with 2. to achieve desired result? – atwellpub Mar 24 '14 at 22:58
  • 1
    POST method helps because "normally" spiders don't follow POST requests when explicitly set. 2. allow to send all the ajax requests to wp-admin, and 'disallow' spiders from 'wp-admin' with robots.txt you protected your admin area and your ajax requests. These 2 methods are best practices, however, gives you no warranty spiders will not follow your admin urls (ajax or not, POST or GET...) they are only a way to discourage spiders. But 3. prevent them, because no spiders can't predict nonces if the nonce isn't in the url. So, to be protected you have to use all three in combination. – gmazzap Mar 24 '14 at 23:36
-1

Here's a programatic method. What it does it checks if an ajax request is running and if the useragent to see if it is in the defined whitelist and if it is then it continues loading wp as normal, else it dies. I used registered_taxonomy because it's the second hook in the hook loading processes:

http://codex.wordpress.org/Plugin_API/Action_Reference

I'm going to add it into the themes functions.php and test:

/* Disable AJAX for Spiders */
add_action( 'registered_taxonomy' , 'disable_ajax_for_spiders' );

function disable_ajax_for_spiders() {

    if ( !defined('DOING_AJAX') || !DOING_AJAX ) {
        return;
    }

    if ( !isset($_SERVER['HTTP_USER_AGENT']) )  {
        return;
    }

    $visitor_useragent = strtolower($_SERVER['HTTP_USER_AGENT']);

    $useragents[] = 'msie';
    $useragents[] = 'firefox';
    $useragents[] = 'webkit';
    $useragents[] = 'opera';
    $useragents[] = 'netscape';
    $useragents[] = 'konqueror';
    $useragents[] = 'gecko';
    $useragents[] = 'chrome';
    $useragents[] = 'songbird';
    $useragents[] = 'seamonkey';
    $useragents[] = 'flock';
    $useragents[] = 'AppleWebKit';
    $useragents[] = 'Android';
    $useragents[] = 'Lynx';
    $useragents[] = 'Dillo';

    /* If useragent in list then bail - else die() immediately*/ 
    if ($visitor_useragent)
    {
        foreach ($useragents as $k=>$useragent)
        {
            $useragent = trim($useragent);
            if ($useragent)
            {
                if(stristr($visitor_useragent, $useragent)||$useragent=='*')
                {
                    return;
                }
            }
        }
    }

    die();
}
  • This is not a solution. The UA string is turned off by some proxies, you don’t send a Vary header, the spiders see a HTTP status 500 now (hurts SEO really bad), and some UAs come with UA strings you forgot. What’s up with Lynx and Dillo? They don’t run AJAX requests. – fuxia Mar 23 '14 at 21:35
  • If we die() an AJAX request it should not show a 500 error message should it? I ran a search on most common useragents and Lnyc and Dillo showed. I am not familiar with either personally. I understand on the proxy traffic. – atwellpub Mar 24 '14 at 23:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.