That filter passes more arguments than you are using.
return apply_filters( 'wp_get_attachment_link', "<a href='$url'>$link_text</a>", $id, $size, $permalink, $icon, $text );
The second of those is the one you need. So you need to alter you callback to use the second parameter, and alter your add_filter
to ask for it by setting the fourth argument to 2.
function add_rel_to_gallery($link, $id) {
return str_replace('<a href=', '<a data-fancybox="true" rel="group-'.$id.'" href=', $link);
}
add_filter( 'wp_get_attachment_link', 'add_rel_to_gallery', 10, 2 );