Return to Revisions

Put your code in the template file category.php.

Remove all the part before the loop: once in category template, you don't need to get the category, get the paged, run again the query with query_posts...

So your category.php should simply appear like so:

if ( have_posts() ) : while ( have_posts() ) : the_post(); 
  $image = wp_get_attachment_image_src( get_post_thumbnail_id( $post->ID ), 'single-post-thumbnail' );
?>

... post markup here

<?php endwhile; 

else: ?>

<p><?php _e('No posts found'); ?></p>

<?php endif; ?>

<!-- Pagination Part -->
<div id="pagination">
  <div class="next"><?php next_posts_link('next &raquo;') ?></div>
  <div class="prev"><?php previous_posts_link('&laquo; previous') ?></div>
</div>

To force that template to show only 4 fosts per page, in functions.php use:

add_action('pre_get_posts','four_post_per_cat');

function four_post_per_cat( $query ) {
  if ( ! is_admin() && is_main_query() && is_category() ) {
    $query->set('posts_per_page', 4);
  }
}

After that numberposts and posts_per_page are synonyms, but numberposts is deprecated. Setting different values for them make numberposts do nothing (or posts_per_page do nothing, I can't remember... however, use one of them).

If your scope is limiting the total posts reached (in all the pages), use the post_limit filter, in functions.php add also:

add_filter( 'post_limits', 'cat_post_limits' );

function cat_post_limits( limit ) {
    return ( is_category() ) ? 'LIMIT 0, 50' : $limit;
}

Following my tips, not also you'll solve your issue, but also improve performace: because query_posts is very bad regarding performance: never use it.

PS: If you get at max 50 posts, 50 is not divisible by 4, so last page will have 2 posts.. why don't set limit to 52 or 48?