Solution for WP-PageNavi with offset?

Question

I have a query with the latest post on a category page. Then another one, with the rest of the posts but with offset=1 so I won't display the same post twice.

I want it this way as the first query with the first post is formatted in some other way and the other ones are different, smaller, etc.

Now, WP-PageNavi doesn't work with offset but there must be a solution. I have tried what I've found on forums but it doesn't work for me.

    // First query with latest post
    <?php query_posts('cat=4&posts_per_page=1'); 
    if(have_posts()) : while(have_posts()) : the_post(); ?>

    <?php the_post_thumbnail('full'); the_title(); echo the_excerpt(); ?>

    <a href="<?php the_permalink(); ?>"> Read story </a>

    <?php endwhile; endif; wp_reset_query(); ?>  

// Second query with the rest of the posts
  <?php query_posts('cat=4&posts_per_page=10&offset=1&paged='.get_query_var('paged')); 
  if(have_posts()) : while(have_posts()) : the_post(); ?>

  <?php the_title(); the_post_thumbnail('medium'); echo the_permalink(); the_excerpt(); ?>

  <?php endwhile; endif;

  if( function_exists('wp_pagenavi')) { wp_pagenavi(); } else { posts_nav_link(); }     
  wp_reset_query(); ?>  

 // I've removed divs from code so it will be easier to read