wp_nav_menu_items filters passes menu arguments as second parameter.
So if you modify code to this:
add_filter( 'wp_nav_menu_items', 'my_custom_menu_item', 10, 2);
function my_custom_menu_item($items, $args) ...
You will be able to check data $args inside function to match specific menu. However the specifics of what to check for depend on how precisely your menus are set up. Might be $args['menu'] holding name, might be $args['theme_location'], etc.
Another option is to simply add your callback right before the needed menu is called and remove right after.