I am working with custom post types and I had trouble with the nav menu not getting the .current-menu-item on my custom post archive page menu item. In fact, when I was on that page or any of the custom posts, instead of the cpt archive menu item getting the current class, the menu item pointing to the articles was getting the .current-menu-item (blog page in my case).

After several hours, google, tests and hacks, I found this thread on wordpress and here's a copy past edited version of the workaround fix:

// As of WP 3.1.1 addition of classes for css styling to parents of custom post types doesn't exist.
// We want the correct classes added to the correct custom post type parent in the wp-nav-menu for css styling and highlighting, so we're modifying each individually...
// The id of each link is required for each one you want to modify
// Place this in your WordPress functions.php file

function remove_parent_classes($class)
{
  // check for current page classes, return false if they exist.
    return ($class == 'current_page_item' || $class == 'current_page_parent' || $class == 'current_page_ancestor'  || $class == 'current-menu-item') ? FALSE : TRUE;
}

function add_class_to_wp_nav_menu($classes)
{
     switch (get_post_type())
     {
        case 'artist':
            // we're viewing a custom post type, so remove the 'current_page_xxx and current-menu-item' from all menu items.
            $classes = array_filter($classes, "remove_parent_classes");

            // add the current page class to a specific menu item (replace ###).
            if (in_array('menu-item-171', $classes))
            {
               $classes[] = 'current_page_parent';
         }
            break;
     }
    return $classes;
}
add_filter('nav_menu_css_class', 'add_class_to_wp_nav_menu');

This solution works when we use the menu item id, but I find it very ugly and it's impossible to use this for plugin development...

Any other idea for selecting the menu item corresponding to the archive page of the custom posts when we are on the archive page or on one of the custom posts in a cleaner way?

In other words, an archive template isn't supposed to select it's own menu item out of the box?

I'd never tought that highliting an element in the wordpress nav menu could have been a problem with custom post types, I'm a bit disapointed. Anyway, here are nnippets of codes I use to manage my custom posts (artist):

artist.php:

/****************************************************************
* Custom Post Type
****************************************************************/

add_action( 'init', 'custom_post_artist' );
function custom_post_artist()
{
  $labels = array(
    [...]
    );

  register_post_type( 'artist',
    array(
      'labels' => $labels,
      'public' => true,
      'menu_position' => 15,
      'supports' => array( 'title', 'editor', 'comments', 'thumbnail', 'revisions', 'excerpt'),
      'show_in_nav_menus' => true,
      'show_in_menu' => true,
      'taxonomies' => array( 'artist_genre', 'artist_music_type' ),
      'has_archive' => 'artistes',
      'rewrite' => array( 'slug' => __('artistes', 'ppdev'), 'with_front' => False )
      )
    );
}

/****************************************************************
* Templates
****************************************************************/

add_filter( 'template_include', 'include_tpl_function', 1 );
function include_tpl_function( $template_path )
{
  if ( get_post_type() == 'artist' )
  {
    if ( is_single() )
    {
      // checks if the file exists in the theme first,
      // otherwise serve the file from the plugin
      if ( $theme_file = locate_template( array('single-artist.php') ) )
      {
        $template_path = $theme_file;
      }
      else
      {
        $template_path = ARTIST_PATH . 'templates/single-artist.php';
      }
    } 
    else if( is_archive() )
    {
      if ( $theme_file = locate_template( array('archive-artist.php') ) )
      {
        $template_path = $theme_file;
      }
      else
      {
        $template_path = ARTIST_PATH . 'templates/archive-artist.php';
      }
    }
  }
  return $template_path;
}

Very basic archive template page:

archive-artist.php:

<section id="primary">
  <div id="content" role="main">
    <?php if ( have_posts() ) : ?>
    <header class="page-header">
      <h1 class="page-title">Latest artists</h1>
    </header>

      <!-- Start the Loop -->         
      <?php while ( have_posts() ) : the_post();

      $thumbnail_attr = array(
        'class' => "aligncenter",
        'alt' => get_the_title()
      );

      <h2>
        <a title="<?php the_title(); ?>" href="<?php the_permalink(); ?>">
          <?php the_title(); ?>
        </a>
      </h2>

      if ( has_post_thumbnail() ) : ?>
          <a title="<?php the_title(); ?>" href="<?php the_permalink(); ?>">
            <?php the_post_thumbnail( 'full', $thumbnail_attr ); ?>
          </a>            
      <?php endif; ?>
      <?php endwhile; ?>
    
     <!-- Display page navigation -->
   <?php global $wp_query;
   if ( isset( $wp_query->max_num_pages ) && $wp_query->max_num_pages > 1 ) { ?>
   <nav id="<?php echo $nav_id; ?>">
     <div class="nav-previous"><?php next_posts_link( '<span class="meta-nav">&larr;</span> Previous artists' ) ); ?></div>
     <div class="nav-next"><?php previous_posts_link( 'Next artists <span class= "meta-nav">&rarr;</span>' ); ?></div>
   </nav>
   <?php };
  endif; ?>
 </div>
</section>

Note: Also tested with twenty twelve theme and same issue