You can only use conditional tags after the posts_selection action hook has run.

According to the codex article on add_feed it should be called with the init action, which runs before. Now I don't know when and how you run it, because you don't say so in the above example, but I'd assume you do it before posts_selection - that would explain your dilemma anyway.

The following is both untested and just guesswork, since you don't really clarify the exact situation in which you want to use the above, but the following ought to work, unless it clashes with the feed actions, with which I have no experience:

function feed_conditional() {
if ( is_home() )
    add_feed( 'home_feed', 'hf_output' );
else
    add_feed( 'regular_feed', 'rf_output' );
}
add_action( 'wp_head', 'feed_conditional' );
// your feed output functions

You can only use conditional tags after the posts_selection action hook has run.

According to the codex article on add_feed it should be called with the init action, which runs before. Now I don't know when and how you run it, because you don't say so in the above example, but I'd assume you do it before posts_selection - that would explain your dilemma anyway.