Unfortunately, that particular filter isn't verified after it's used. Here is the use of that filter in core:
extract( apply_filters( 'wp_mail', compact( 'to', 'subject', 'message', 'headers', 'attachments' ) ) );
So all that filter does is populate the $to, $subject, $message, $headers, and $attechments varaibles. It's not an action hook, so while you probably could throw some kind of termination operation in there, you really shouldn't. Ideally, you'd be able to return false from your filtering function to kill operation, but the function isn't set up that way.
Instead, I recommend hooking to the phpmailer_init action. It's the last action in the wp_mail() function and it passes a reference to the actual $phpmailer object that does the mailing.
This untested function should prevent mail from sending:
class fakemailer {
public function Send() {
throw new phpmailerException( 'Cancelling mail' );
}
}
if ( ! class_exists( 'phpmailerException' ) ) :
class phpmailerException extends Exception {
public function errorMessage() {
$errorMsg = '<strong>' . $this->getMessage() . "</strong><br />\n";
return $errorMsg;
}
}
add_action( 'phpmailer_init', 'wpse_53612_fakemailer' );
function wpse_53612_fakemailer( $phpmailer ) {
if ( ! /* condition */ )
$phpmailer = new fakemailer();
}
This should replace the $phpmailer object with an instance of your fake mailer class. This fake class only contains a Send() method that immediately throws an exception of type phpmailerException. The wp_mail() function will catch this exception and return false by default.
Not the most performant solution in the world ... you should really be checking conditions before even calling wp_mail() (as suggested by @Zaidar), but if you must use a hook, this is one way to do it.