In a multi-site installation it is 'wpmu_validate_user_signup'
:
add_filter( 'wpmu_validate_user_signup', 'wpse_77904_mu_no_at_in_username' );
function wpse_77904_mu_no_at_in_username( $user )
{
if ( FALSE !== strpos( $user['user_name'], '@' ) )
$user['errors']->add('user_name', __( 'That username is not allowed.' ) );
return $user;
}
On single-site it is 'validate_username'
:
add_filter( 'validate_username', 'wpse_77904_no_at_in_username', 10, 2 );
function wpse_77904_no_at_in_username( $valid, $username )
{
if ( FALSE !== strpos( $username, '@' ) )
return FALSE;
return $valid;
}
Caveat: both not tested. :)