WordPress Development Stack Exchange is a question and answer site for WordPress developers and administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Does anyone know how I can have a unique robots.txt file for every domain served by my wp-multisite-installation? I did a search on plugins but couldn't find anything suitable.

share|improve this question

Straight from the source, (line 1845 wp-includes/functions.php, 3.3.1):

function do_robots() {
    header( 'Content-Type: text/plain; charset=utf-8' );

    do_action( 'do_robotstxt' );

    $output = "User-agent: *\n";
    $public = get_option( 'blog_public' );
    if ( '0' == $public ) {
        $output .= "Disallow: /\n";
    } else {
        $site_url = parse_url( site_url() );
        $path = ( !empty( $site_url['path'] ) ) ? $site_url['path'] : '';
        $output .= "Disallow: $path/wp-admin/\n";
        $output .= "Disallow: $path/wp-includes/\n";

    echo apply_filters('robots_txt', $output, $public);

So to customise it:

function my_custom_robots( $robots )
    if ( my_condition() )
        $robots .= "\nDisallow: /something/else/";
    return $robots;
add_filter( 'robots_txt', 'my_custom_robots' );
share|improve this answer
Thanks for this. A lot of blog posts just talk about the filters but yours is the first that demonstrates by actually using it. – racl101 Nov 22 '12 at 16:42

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.