[That filter passes more arguments than you are using][1].

    return apply_filters( 'wp_get_attachment_link', "<a href='$url'>$link_text</a>", $id, $size, $permalink, $icon, $text );

The second of those is the one you need. So you need to alter you callback to use the second parameter, and alter your `add_filter` to ask for it by setting the fourth argument to 2.

    function add_rel_to_gallery($link, $id) {
      return str_replace('<a href=', '<a data-fancybox="true" rel="group-'.$id.'" href=', $link);
    }
    add_filter( 'wp_get_attachment_link', 'add_rel_to_gallery', 10, 2 );

After seeing your edit, I realized that I misread what you were attempting and I tried to find a way to create a static gallery specific ID without abusing hooks too much. I am not sure I succeeded in that last part but the following seems to work:

    function get_id($inc = false) {
      static $id;
      if ($inc) {
    	$id++;
      }
      return $id;
    }
    
    function replace($link) {
      $id = get_id();
      return str_replace('<a href=', '<a data-fancybox="true" rel="group-'.$id.'" href=', $link);
    }
    
    add_filter(
      'post_gallery',
      function() {
    	get_id(true);
    	add_filter('wp_get_attachment_link','replace');
      }
    );

The correct answer is probably to use the [`post_gallery` hook][2] to hijack the rest of the gallery shortcode, or [just overwrite the gallery shortcode with one of your own][3].


  [1]: https://core.trac.wordpress.org/browser/tags/3.8.1/src/wp-includes/post-template.php#L1207
  [2]: https://core.trac.wordpress.org/browser/tags/3.5.1/wp-includes/media.php#L685
  [3]: http://wordpress.stackexchange.com/a/100345/21376