[That filter passes more arguments than you are using][1].

    return apply_filters( 'wp_get_attachment_link', "<a href='$url'>$link_text</a>", $id, $size, $permalink, $icon, $text );

The second of those is the one you need. So you need to alter you callback to use the second parameter, and alter your `add_filter` to ask for it by setting the fourth argument to 2.

    function add_rel_to_gallery($link, $id) {
      return str_replace('<a href=', '<a data-fancybox="true" rel="group-'.$id.'" href=', $link);
    }
    add_filter( 'wp_get_attachment_link', 'add_rel_to_gallery', 10, 2 );


  [1]: https://core.trac.wordpress.org/browser/tags/3.8.1/src/wp-includes/post-template.php#L1207