This is a fairly new installation. Almost no plugins. Everyting works fine except the dinamically-generated robots.txt gives Error 500. Google doesn't like it much. (won't read my sitemap).

All other pages work fine.

Permalinks are "default".

htaccess on public_html is:

# BEGIN WordPress

# END WordPress

Server logs are useless: - - [11/Dec/2012:15:06:11 -0500] "GET /robots.txt HTTP/1.1" 500 671 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

I'm definitely no WP expert. Found function do_robots() in functions.php but don't know who calls it.

I'd appreciate any pointers on how to troubleshoot this further. Otherwise... maybe generating a static robots.txt file? Not sure how would that interact wit the one WP tries to build.

EDIT: As requested, this is the function in my functions.php file:

 * Display the robots.txt file content.
 * The echo content should be with usage of the permalinks or for creating the
 * robots.txt file.
 * @since 2.1.0
 * @uses do_action() Calls 'do_robotstxt' hook for displaying robots.txt rules.

function do_robots() { 
    header( 'Content-Type: text/plain; charset=utf-8' );

    do_action( 'do_robotstxt' );

    $output = "User-agent: *\n";
    $public = get_option( 'blog_public' );
    if ( '0' == $public ) {
        $output .= "Disallow: /\n";
    } else {
        $site_url = parse_url( site_url() );
        $path = ( !empty( $site_url['path'] ) ) ? $site_url['path'] : '';
        $output .= "Disallow: $path/wp-admin/\n";
        $output .= "Disallow: $path/wp-includes/\n";

    echo apply_filters('robots_txt', $output, $public);
  • What is your robots.txt permission? Keept it 644
    – Lenin
    Commented Dec 11, 2012 at 20:22
  • Does your WP work without problem?
    – Lenin
    Commented Dec 11, 2012 at 20:23
  • WP works fine. There is no actual robots.txt file. WP should generate it dinamically
    – Henry
    Commented Dec 11, 2012 at 20:28
  • Without it WP should work, remove/disable the function and see what happens.
    – The Alpha
    Commented Dec 11, 2012 at 20:29
  • Are you sure it would be created on request? I dont think so. Your robots.txt generator has problem.
    – Lenin
    Commented Dec 11, 2012 at 20:30

3 Answers 3


1) Don't alter or edit core Wordpress files or folders.

2) There is no function in Wordpress core that dynamically generates a robots.txt file. The theme you are using is trying to create the robots.txt file with that function in functions.php and must be creating it with the wrong character encoding, and that is causing the 500 Server Error.

3) Disable that function - ask the theme developer how - and make your own robots.txt file with an FTP client with the character encoding of Unicode-no BOM.

The functions do_robotstxt runs in the do_robots function before it prints out the Disallow lists for the robots.txt file. http://adambrown.info/p/wp_hooks/hook/do_robotstxt

It's the function Wordpress uses to block indexing of sites. The theme developer has "hijacked" the function incorrectly.

A sample robots.txt file for Wordpress sites from http://codex.wordpress.org/Search_Engine_Optimization_for_WordPress

Sitemap: http://www.example.com/sitemap.xml

# Google Image
User-agent: Googlebot-Image
Allow: /*

# Google AdSense
User-agent: Mediapartners-Google*

# digg mirror
User-agent: duggmirror
Disallow: /

# global
User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/cache/
Disallow: /wp-content/themes/
Disallow: /trackback/
Disallow: /feed/
Disallow: /comments/
Disallow: /category/*/*
Disallow: */trackback/
Disallow: */feed/
Disallow: */comments/
Disallow: /*?
Allow: /wp-content/uploads/
  • 1) of course. any editing i may do is just temporary for debugging. 2) there is a function called do_robots in wp-includes/functions.php. Since that file is not in the themes or plugins folders I assumed it was original from WP. 3) the theme is from here: http://themes.bavotasan.com/ 4)I made a manual robots.txt file. That solves the immediate issue: Google now reads my sitemap.xml. Still would be nice to get to the root of the problem. Thanks for your help
    – Henry
    Commented Dec 11, 2012 at 22:38
  • Good that it worked; I added a sample robots.txt file. The original problem might be that Content-Type: text/plain; charset=utf-8 in the function, or something on your server; windows or linux? Commented Dec 11, 2012 at 22:58
  • Linux, standard installation through cPanel, with all defaults. Thanks for the sample file.
    – Henry
    Commented Dec 11, 2012 at 23:04
  • Actually, the function rewrite_rules() in wp-includes\rewrite.php seems to be handling rewrites for robots.txt and it eventually gets routed through: do_robots.
    – Kevin
    Commented May 6, 2013 at 16:20

500 internal server error is given to hide actual error message for security reasons.

Your robots.txt is not generated and the function has a problem which throws error and thats the reason you are seeing this error.


Sounds like it is not related to robots.txt generation but some other functionality. Try to turn off all plugins and revert to the default theme. If at that point robots.txt is accessible then start activating the plugins and after each activation test robots.txt.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.