Take the 2-minute tour ×
WordPress Development Stack Exchange is a question and answer site for WordPress developers and administrators. It's 100% free, no registration required.

This is a fairly new installation. Almost no plugins. Everyting works fine except the dinamically-generated robots.txt gives Error 500. Google doesn't like it much. (won't read my sitemap).

All other pages work fine.

Permalinks are "default".

htaccess on public_html is:

# BEGIN WordPress

# END WordPress

Server logs are useless:

66.249.73.225 - - [11/Dec/2012:15:06:11 -0500] "GET /robots.txt HTTP/1.1" 500 671 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

I'm definitely no WP expert. Found function do_robots() in functions.php but don't know who calls it.

I'd appreciate any pointers on how to troubleshoot this further. Otherwise... maybe generating a static robots.txt file? Not sure how would that interact wit the one WP tries to build.

EDIT: As requested, this is the function in my functions.php file:

/**
 * Display the robots.txt file content.
 *
 * The echo content should be with usage of the permalinks or for creating the
 * robots.txt file.
 *
 * @since 2.1.0
 * @uses do_action() Calls 'do_robotstxt' hook for displaying robots.txt rules.
 */

function do_robots() { 
    header( 'Content-Type: text/plain; charset=utf-8' );

    do_action( 'do_robotstxt' );

    $output = "User-agent: *\n";
    $public = get_option( 'blog_public' );
    if ( '0' == $public ) {
        $output .= "Disallow: /\n";
    } else {
        $site_url = parse_url( site_url() );
        $path = ( !empty( $site_url['path'] ) ) ? $site_url['path'] : '';
        $output .= "Disallow: $path/wp-admin/\n";
        $output .= "Disallow: $path/wp-includes/\n";
    }

    echo apply_filters('robots_txt', $output, $public);
}
share|improve this question
    
What is your robots.txt permission? Keept it 644 –  Lenin Dec 11 '12 at 20:22
    
Does your WP work without problem? –  Lenin Dec 11 '12 at 20:23
    
WP works fine. There is no actual robots.txt file. WP should generate it dinamically –  Henry Dec 11 '12 at 20:28
    
Without it WP should work, remove/disable the function and see what happens. –  Sheikh Heera Dec 11 '12 at 20:29
    
Are you sure it would be created on request? I dont think so. Your robots.txt generator has problem. –  Lenin Dec 11 '12 at 20:30
show 4 more comments

migrated from stackoverflow.com Dec 12 '12 at 3:16

This question came from our site for professional and enthusiast programmers.

3 Answers

1) Don't alter or edit core Wordpress files or folders.

2) There is no function in Wordpress core that dynamically generates a robots.txt file. The theme you are using is trying to create the robots.txt file with that function in functions.php and must be creating it with the wrong character encoding, and that is causing the 500 Server Error.

3) Disable that function - ask the theme developer how - and make your own robots.txt file with an FTP client with the character encoding of Unicode-no BOM.

The functions do_robotstxt runs in the do_robots function before it prints out the Disallow lists for the robots.txt file. http://adambrown.info/p/wp_hooks/hook/do_robotstxt

It's the function Wordpress uses to block indexing of sites. The theme developer has "hijacked" the function incorrectly.

A sample robots.txt file for Wordpress sites from http://codex.wordpress.org/Search_Engine_Optimization_for_WordPress

Sitemap: http://www.example.com/sitemap.xml

# Google Image
User-agent: Googlebot-Image
Disallow:
Allow: /*

# Google AdSense
User-agent: Mediapartners-Google*
Disallow:

# digg mirror
User-agent: duggmirror
Disallow: /

# global
User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/cache/
Disallow: /wp-content/themes/
Disallow: /trackback/
Disallow: /feed/
Disallow: /comments/
Disallow: /category/*/*
Disallow: */trackback/
Disallow: */feed/
Disallow: */comments/
Disallow: /*?
Allow: /wp-content/uploads/
share|improve this answer
    
1) of course. any editing i may do is just temporary for debugging. 2) there is a function called do_robots in wp-includes/functions.php. Since that file is not in the themes or plugins folders I assumed it was original from WP. 3) the theme is from here: http://themes.bavotasan.com/ 4)I made a manual robots.txt file. That solves the immediate issue: Google now reads my sitemap.xml. Still would be nice to get to the root of the problem. Thanks for your help –  Henry Dec 11 '12 at 22:38
    
Good that it worked; I added a sample robots.txt file. The original problem might be that Content-Type: text/plain; charset=utf-8 in the function, or something on your server; windows or linux? –  songdogtech Dec 11 '12 at 22:58
    
Linux, standard installation through cPanel, with all defaults. Thanks for the sample file. –  Henry Dec 11 '12 at 23:04
    
Actually, the function rewrite_rules() in wp-includes\rewrite.php seems to be handling rewrites for robots.txt and it eventually gets routed through: do_robots. –  Kevin May 6 '13 at 16:20
add comment

500 internal server error is given to hide actual error message for security reasons.

Your robots.txt is not generated and the function has a problem which throws error and thats the reason you are seeing this error.

share|improve this answer
add comment

Sounds like it is not related to robots.txt generation but some other functionality. Try to turn off all plugins and revert to the default theme. If at that point robots.txt is accessible then start activating the plugins and after each activation test robots.txt.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.