Wordpress: robots.txt gives 500 Internal Server Error

Question

This is a fairly new installation. Almost no plugins. Everyting works fine except the dinamically-generated robots.txt gives Error 500. Google doesn't like it much. (won't read my sitemap).

All other pages work fine.

Permalinks are "default".

htaccess on public_html is:

# BEGIN WordPress

# END WordPress

Server logs are useless:

66.249.73.225 - - [11/Dec/2012:15:06:11 -0500] "GET /robots.txt HTTP/1.1" 500 671 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

I'm definitely no WP expert. Found function do_robots() in functions.php but don't know who calls it.

I'd appreciate any pointers on how to troubleshoot this further. Otherwise... maybe generating a static robots.txt file? Not sure how would that interact wit the one WP tries to build.

EDIT: As requested, this is the function in my functions.php file:

/**
 * Display the robots.txt file content.
 *
 * The echo content should be with usage of the permalinks or for creating the
 * robots.txt file.
 *
 * @since 2.1.0
 * @uses do_action() Calls 'do_robotstxt' hook for displaying robots.txt rules.
 */

function do_robots() { 
    header( 'Content-Type: text/plain; charset=utf-8' );

    do_action( 'do_robotstxt' );

    $output = "User-agent: *\n";
    $public = get_option( 'blog_public' );
    if ( '0' == $public ) {
        $output .= "Disallow: /\n";
    } else {
        $site_url = parse_url( site_url() );
        $path = ( !empty( $site_url['path'] ) ) ? $site_url['path'] : '';
        $output .= "Disallow: $path/wp-admin/\n";
        $output .= "Disallow: $path/wp-includes/\n";
    }

    echo apply_filters('robots_txt', $output, $public);
}

Answer 1

1) Don't alter or edit core Wordpress files or folders.

2) There is no function in Wordpress core that dynamically generates a robots.txt file. The theme you are using is trying to create the robots.txt file with that function in functions.php and must be creating it with the wrong character encoding, and that is causing the 500 Server Error.

3) Disable that function - ask the theme developer how - and make your own robots.txt file with an FTP client with the character encoding of Unicode-no BOM.

The functions do_robotstxt runs in the do_robots function before it prints out the Disallow lists for the robots.txt file. http://adambrown.info/p/wp_hooks/hook/do_robotstxt

It's the function Wordpress uses to block indexing of sites. The theme developer has "hijacked" the function incorrectly.

A sample robots.txt file for Wordpress sites from http://codex.wordpress.org/Search_Engine_Optimization_for_WordPress

Sitemap: http://www.example.com/sitemap.xml

# Google Image
User-agent: Googlebot-Image
Disallow:
Allow: /*

# Google AdSense
User-agent: Mediapartners-Google*
Disallow:

# digg mirror
User-agent: duggmirror
Disallow: /

# global
User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/cache/
Disallow: /wp-content/themes/
Disallow: /trackback/
Disallow: /feed/
Disallow: /comments/
Disallow: /category/*/*
Disallow: */trackback/
Disallow: */feed/
Disallow: */comments/
Disallow: /*?
Allow: /wp-content/uploads/

Answer 2

500 internal server error is given to hide actual error message for security reasons.

Your robots.txt is not generated and the function has a problem which throws error and thats the reason you are seeing this error.

Answer 3

Sounds like it is not related to robots.txt generation but some other functionality. Try to turn off all plugins and revert to the default theme. If at that point robots.txt is accessible then start activating the plugins and after each activation test robots.txt.