Return to Answer

If you want limit the number of items in a gallery, you can use:

[gallery maxitems="50"]

where the custom items attribute is the maximum number of items to display.

This attribute is supported by the following plugin:

<?php
/**
 * Plugin Name:   The Gallery Shortcode - Maximum Number Of Items.
 * Description:   Support the 'maxitems' attribute in the gallery shortcode
 * Plugin URI:    http://wordpress.stackexchange.com/a/176937/26350
 * Plugin Author: Birgir Erlendsson (birgire)
 * Version:       0.0.2
 */

add_action( 'wp', function()
{
    if( class_exists( 'GalleryMaxItems' ) )
    {
        $o = new GalleryMaxItems;
        $o->init();
    }
} );

class GalleryMaxItems
{
    private $maxitems;

    public function init()
    {
        add_filter( 'post_gallery', array( $this, 'post_gallery' ), 10, 3 );
    }

    public function post_gallery( $output = '', $attr = array(), $instance = 0 )
    {
        if( isset( $attr['maxitems'] ) )
        {
            $this->maxitems = $attr['maxitems'];
            add_filter( 'query', array( $this, 'query' ) );
        }
        return $output;
    }

    public function query( $sql )
    {
        remove_filter( current_filter(), array( $this, __FUNCTION__ ) );

        $limit  = sprintf( " LIMIT 0, %d ", $this->maxitems );
        $regex  = '#LIMIT 0,\s\d+#';

        if( false !== stripos( $sql, ' LIMIT ' ) )
            $sql = preg_replace( $regex, $limit, $sql );
         else
            $sql .= $limit; 

        return $sql;
    }

} // end class

If you want limit the number of items in a gallery, there seems to be a plugin available on wordpress.org called Limit parameter for gallery shortcode (no affiliation). It uses the pre_get_posts filter to change the posts_per_page and offset attributes of the get_posts() or get_children() calls in the gallery shortcode callback.

Example:

[gallery id="4" limit="3" offset="1"]

<?php
/**
 * Plugin Name:   The Gallery Shortcode - Maximum Number Of Items.
 * Description:   Support the 'maxitems' attribute in the gallery shortcode
 * Plugin URI:    http://wordpress.stackexchange.com/a/176937/26350
 * Plugin Author: Birgir Erlendsson (birgire)
 * Version:       0.0.2
 */

add_action( 'wp', function()
{
    if( class_exists( 'GalleryMaxItems' ) )
    {
        $o = new GalleryMaxItems;
        $o->init();
    }
} );

class GalleryMaxItems
{
    private $maxitems;

    public function init()
    {
        add_filter( 'post_gallery', array( $this, 'post_gallery' ), 10, 3 );
    }

    public function post_gallery( $output = '', $attr = array(), $instance = 0 )
    {
        if( isset( $attr['maxitems'] ) )
        {
            $this->maxitems = $attr['maxitems'];
            add_filter( 'query', array( $this, 'query' ) );
        }
        return $output;
    }

    public function query( $sql )
    {
         remove_filter( current_filter(), array( $this, __FUNCTION__ ) );

         $limit = sprintf( " LIMIT 0, %d ", $this->maxitems );
         $regex = '#LIMIT 0,\s\d+#';

         if( false !== stripos( $sql, ' LIMIT ' ) )
             $sql = preg_replace( $regex, $limit, $sql );
          else
             $sql .= $limit; 

     return $sql;
}

} // end class

Few remarks:

One possible solution is to override the gallery shortcode callback, but I don't recommend that, in case it will change in the future.

Since the gallery shortcode is using get_posts() or the get_children() wrapper, to fetch the gallery images, there's no direct way to filter those queries. Not even to change the suppress_filters attribute.

First I wrote a solution that chops down the number of images in the ids/include attributes, but that will not work when the id attribute is set.

For that case I wrote a solution that just duplicated the get_children() query, in the gallery source code. Then we could generate a new chopped list of ids, to include again in the gallery_shortcode() function via the post_gallery filter. But I didn't like this solution since then we had to query the images twice!

Therefore I choose the query filter to directly modify the LIMIT of the generated SQL query. This should work no matter what combination of shortcode attributes you use.

<?php
/**
 * Plugin Name:   The Gallery Shortcode - Maximum Number Of Items.
 * Description:   Support the 'maxitems' attribute in the gallery shortcode
 * Plugin URI:    http://wordpress.stackexchange.com/a/176937/26350
 * Plugin Author: Birgir Erlendsson (birgire)
 * Version:       0.0.2
 */

add_action( 'wp', function()
{
    if( class_exists( 'GalleryMaxItems' ) )
    {
        $o = new GalleryMaxItems;
        $o->init();
    }
} );

class GalleryMaxItems
{
    private $maxitems;

    public function init()
    {
        add_filter( 'post_gallery', array( $this, 'post_gallery' ), 10, 3 );
    }

    public function post_gallery( $output = '', $attr = array(), $instance = 0 )
    {
        if( isset( $attr['maxitems'] ) )
        {
            $this->maxitems = $attr['maxitems'];
            add_filter( 'query', array( $this, 'query' ) );
        }
        return $output;
    }

    public function query( $sql )
    {
        remove_filter( current_filter(), array( $this, __FUNCTION__ ) );

        $limit  = sprintf( " LIMIT 0, %d ", $this->maxitems );
        $regex  = '#LIMIT 0,\s\d+#';

        if( false !== stripos( $sql, ' LIMIT ' ) )
            $sql = preg_replace( $regex, $limit, $sql );
         else
            $sql .= $limit; 

        return $sql;
    }

} // end class

<?php
/**
 * Plugin Name:   The Gallery Shortcode - Maximum Number Of Items.
 * Description:   Support the 'maxitems' attribute in the gallery shortcode
 * Plugin URI:    http://wordpress.stackexchange.com/a/176937/26350
 * Plugin Author: Birgir Erlendsson (birgire)
 * Version:       0.0.1
 */

add_action( 'wp', function()
{
    if( class_exists( 'GalleryMaxItems' ) )
    {
        $o = new GalleryMaxItems;
        $o->init();
    }
} );

class GalleryMaxItems
{
    private $maxitems;

    public function init()
    {
        add_filter( 'post_gallery', array( $this, 'post_gallery' ), 10, 3 );
    }

    public function post_gallery( $output = '', $attr = array(), $instance = 0 )
    {
        if( isset( $attr['maxitems'] ) )
        {
            $this->maxitems = $attr['maxitems'];
            add_filter( 'query', array( $this, 'query' ) );
        }
        return $output;
    }

    public function query( $sql )
    {
         remove_filter( current_filter(), array( $this, __FUNCTION__ ) );
         return preg_replace( '#LIMIT 0,\s\d+#', sprintf( "LIMIT 0, %d", $this->maxitems ), $sql );
    }

} // end class

<?php
/**
 * Plugin Name:   The Gallery Shortcode - Maximum Number Of Items.
 * Description:   Support the 'maxitems' attribute in the gallery shortcode
 * Plugin URI:    http://wordpress.stackexchange.com/a/176937/26350
 * Plugin Author: Birgir Erlendsson (birgire)
 * Version:       0.0.2
 */

add_action( 'wp', function()
{
    if( class_exists( 'GalleryMaxItems' ) )
    {
        $o = new GalleryMaxItems;
        $o->init();
    }
} );

class GalleryMaxItems
{
    private $maxitems;

    public function init()
    {
        add_filter( 'post_gallery', array( $this, 'post_gallery' ), 10, 3 );
    }

    public function post_gallery( $output = '', $attr = array(), $instance = 0 )
    {
        if( isset( $attr['maxitems'] ) )
        {
            $this->maxitems = $attr['maxitems'];
            add_filter( 'query', array( $this, 'query' ) );
        }
        return $output;
    }

    public function query( $sql )
    {
         remove_filter( current_filter(), array( $this, __FUNCTION__ ) ); 

         $limit = sprintf( " LIMIT 0, %d ", $this->maxitems );
         $regex = '#LIMIT 0,\s\d+#';

         if( false !== stripos( $sql, ' LIMIT ' ) )
             $sql = preg_replace( $regex, $limit, $sql );
          else
             $sql .= $limit; 

     return $sql;
}

} // end class