Customizing the menu this way is as for as I can tell, not very friendly. What I had to do recently was custom built the submenu item using remove_submenu_page for all the sub menu items and then add them in the order I wanted using add_submenu_page.

A simple example

 //this was first sub-menu item I did not want it #1
 remove_submenu_page( 'edit.php?post_type=test', 'edit.php?post_type=test' );

 // add the actual sub-menu item I wanted as #1
 add_submenu_page('edit.php?post_type=hello','Hello', 'Hello', 'manage_options', 'my_hello', 'my_hello_options');

//re-add the orginal #1 item removed above so it is now the #2 item
 add_submenu_page('edit.php?post_type=test','Edit Test', 'Edit Test', 'manage_options', 'edit.php?post_type=test');

Customizing the menu this way is as for as I can tell, not very friendly. What I had to do recently was custom built the submenu item using remove_submenu_page for all the sub menu items and then add them in the order I wanted using add_submenu_page.

A simple example

 //this was first sub-menu item I did not want it #1
 remove_submenu_page( 'edit.php?post_type=test', 'edit.php?post_type=test' );

 // add the actual sub-menu item I wanted as #1
 add_submenu_page('edit.php?post_type=hello','Hello', 'Hello', 'manage_options', 'my_hello', 'my_hello_options');

//re-add the orginal #1 item removed above so it is now the #2 item
 add_submenu_page('edit.php?post_type=test','Edit Test', 'Edit Test', 'manage_options', 'edit.php?post_type=test');

Edit: If you plan on using the menu_order filter, it's pretty simple to get the filter working: Just add this line before it. Else it would skip the menu_order filter: add_filter( 'custom_menu_order', '__return_true' );.