When I want to redirect the user after successful registration, I just use "registration_redirect" filter hook. How about when registration fails, which filter hook should I use? I looked into wp-login.php but I didn't see any filter/action hook that I could use. How do I go about this?
You could add an htaccess redirect or try out the theme my login plugin: http://wordpress.org/extend/plugins/theme-my-login/. It has options for custom redirection.
There is an more sophisticated solution instead of htaccess or plugin. For those interested in this topic, there's my code to redirect after (exactly before) unsuccessfull resitration attempt.
Why before? Before an attempt for registration, WordPress checks for an username and email address, whether they are not empty, whether they are not already in DB, and whether email nor username are in correct. If so, it reports errors and you can hook to a function right after this check.
Next step - unhookable - is an attempt to create a new user. If this fails, that there is an another error message and you'll be redirected to standard wp_login.php screen.
But, there is an but. This process, Next step, just calls wp_insert_user() and this function fails only on empty_user_login, existing_user_login and existing_user_email - otherwise it end successfully. And if you've read carefully, all these variables are checked before this function is called (this double check is logical as you can call wp_insert_user directly from your plugin).
So if register process fails after mentioned hook, it fails because there's something wrong with your WordPress installation and so, it is not a fail of your redirection function hooked to register process.
And finally some code for those, ho just want to copy and paste a code for redirection to custom page after login fails: