Short of de-registering the built in shortcode tag, copying the function and re-registering it with some modifications you could filter on wp_get_attachment_link. The problem is this affects everywhere that function is used.
Saying that it might not be such a bad thing:
add_filter( 'wp_get_attachment_link', 'my_attachment_link', 10, 6 );
function my_attachment_link( $link, $id, $size, $permalink, $icon, $text ) {
if ( !is_admin() ) {
// if we're linking to the file itself
if ( ! $permalink )
$link = preg_replace("/href=\'([^\']+)\'/", "href='" . wp_get_attachment_image_src( $id, 'my-custom-size', false ) . "'", $link );
}
return $link;
}
This will override any links generated to the original size file on the front end with a link to the size you specify in the second argument in wp_get_attachment_image_src().