I'm reposting this with an example, I really don't get the code formatting of this editor, last time I was so irritated with it that I skipped the code!
That dropdown is populated using a filter 'wp_dropdown_users' (user.php, line 976). This filter returns the dropdown (html select) as a string. You can intercept this string, and add your own options, which is the list of users with the custom role. Inspect the select with Firebug, the option value is the user id, and text is the login name for that user.
//Doing it only for the custom post type
if($post->post_type == 'my_custom_post')
$users = get_users(array('role'=>'my_custom_role'));
//We're forming a new select with our values, you can add an option
//with value 1, and text as 'admin' if you want the admin to be listed as well,
//optionally you can use a simple string replace trick to insert your options,
//if you don't want to override the defaults
$output .= "<select id='post_author_override' name='post_author_override' class=''>";
foreach($users as $user)
$output .= "<option value='".$user->id."'>".$user->user_login."</option>";
$output .= "</select>";
That's it! You'll have the dropdown listing your custom roles. Try changing the post author, it gets updated neatly. It's a hack, but it worked for me!