This should get you going in the right direction. Not my solution, slightly modified from here.

You'll need to swap out your-post-type for the post type you're using.

function adjust_post_formats() {
    if ( $post_id ) {
        $post = get_post( $post_id );
        if ( $post )
            $post_type = $post->post_type;
    } elseif ( !isset($_GET['post_type']) )
        $post_type = 'post';
    elseif ( in_array( $_GET['post_type'], get_post_types( array('show_ui' => true ) ) ) )
        $post_type = $_GET['post_type'];
    else
        return;

    if ( 'your-post-type' == $post_type )
        add_theme_support( 'post-formats', array( 'gallery' ) );
    elseif ( 'post' == $post_type )
        add_theme_support( 'post-formats', array( 'gallery', 'link', 'quote', 'audio', 'video' ) );
}
add_action( 'load-post.php','adjust_post_formats' );
add_action( 'load-post-new.php','adjust_post_formats' );

This should get you going in the right direction. Not my solution, slightly modified from here.

You'll need to swap out your-post-type for the post type you're using.

function adjust_post_formats() {
    if (isset($_GET['post'])) {
        $post = get_post($_GET['post']);
        if ($post)
            $post_type = $post->post_type;
    } elseif ( !isset($_GET['post_type']) )
        $post_type = 'post';
    elseif ( in_array( $_GET['post_type'], get_post_types( array('show_ui' => true ) ) ) )
        $post_type = $_GET['post_type'];
    else
        return;

    if ( 'your-post-type' == $post_type )
        add_theme_support( 'post-formats', array( 'gallery' ) );
    elseif ( 'post' == $post_type )
        add_theme_support( 'post-formats', array( 'gallery', 'link', 'quote', 'audio', 'video' ) );
}
add_action( 'load-post.php','adjust_post_formats' );
add_action( 'load-post-new.php','adjust_post_formats' );