That filter passes more arguments than you are using.

return apply_filters( 'wp_get_attachment_link', "<a href='$url'>$link_text</a>", $id, $size, $permalink, $icon, $text );

The second of those is the one you need. So you need to alter you callback to use the second parameter, and alter your add_filter to ask for it by setting the fourth argument to 2.

function add_rel_to_gallery($link, $id) {
  return str_replace('<a href=', '<a data-fancybox="true" rel="group-'.$id.'" href=', $link);
}
add_filter( 'wp_get_attachment_link', 'add_rel_to_gallery', 10, 2 );

After seeing your edit, I realized that I misread what you were attempting and I tried to find a way to create a static gallery specific ID without abusing hooks too much. I am not sure I succeeded in that last part but the following seems to work:

function get_id($inc = false) {
  static $id;
  if ($inc) {
    $id++;
  }
  return $id;
}

function replace($link) {
  $id = get_id();
  return str_replace('<a href=', '<a data-fancybox="true" rel="group-'.$id.'" href=', $link);
}

add_filter(
  'post_gallery',
  function() {
    get_id(true);
    add_filter('wp_get_attachment_link','replace');
  }
);

The correct answer is probably to use the post_gallery hook to hijack the rest of the gallery shortcode, or just overwrite the gallery shortcode with one of your own.

That filter passes more arguments than you are using.

return apply_filters( 'wp_get_attachment_link', "<a href='$url'>$link_text</a>", $id, $size, $permalink, $icon, $text );

The second of those is the one you need. So you need to alter you callback to use the second parameter, and alter your add_filter to ask for it by setting the fourth argument to 2.

function add_rel_to_gallery($link, $id) {
  return str_replace('<a href=', '<a data-fancybox="true" rel="group-'.$id.'" href=', $link);
}
add_filter( 'wp_get_attachment_link', 'add_rel_to_gallery', 10, 2 );

After seeing your edit, I realized that I misread what you were attempting and I tried to find a way to create a static gallery specific ID without abusing hooks too much. I am not sure I succeeded in that last part but the following seems to work:

function get_id($inc = false) {
  static $id;
  if ($inc) {
    $id++;
  }
  return $id;
}

function replace($link) {
  $id = get_id();
  return str_replace('<a href=', '<a data-fancybox="true" rel="group-'.$id.'" href=', $link);
}

add_filter(
  'post_gallery',
  function() {
    get_id(true);
    add_filter('wp_get_attachment_link','replace');
  }
);

The correct answer is probably to use the post_gallery hook to hijack the rest of the gallery shortcode, or just overwrite the gallery shortcode with one of your own.

That filter passes more arguments than you are using.

return apply_filters( 'wp_get_attachment_link', "<a href='$url'>$link_text</a>", $id, $size, $permalink, $icon, $text );

The second of those is the one you need. So you need to alter you callback to use the second parameter, and alter your add_filter to ask for it by setting the fourth argument to 2.

function add_rel_to_gallery($link, $id) {
  return str_replace('<a href=', '<a data-fancybox="true" rel="group-'.$id.'" href=', $link);
}
add_filter( 'wp_get_attachment_link', 'add_rel_to_gallery', 10, 2 );

That filter passes more arguments than you are using.

return apply_filters( 'wp_get_attachment_link', "<a href='$url'>$link_text</a>", $id, $size, $permalink, $icon, $text );

The second of those is the one you need. So you need to alter you callback to use the second parameter, and alter your add_filter to ask for it by setting the fourth argument to 2.

function add_rel_to_gallery($link, $id) {
  return str_replace('<a href=', '<a data-fancybox="true" rel="group-'.$id.'" href=', $link);
}
add_filter( 'wp_get_attachment_link', 'add_rel_to_gallery', 10, 2 );

After seeing your edit, I realized that I misread what you were attempting and I tried to find a way to create a static gallery specific ID without abusing hooks too much. I am not sure I succeeded in that last part but the following seems to work:

function get_id($inc = false) {
  static $id;
  if ($inc) {
    $id++;
  }
  return $id;
}

function replace($link) {
  $id = get_id();
  return str_replace('<a href=', '<a data-fancybox="true" rel="group-'.$id.'" href=', $link);
}

add_filter(
  'post_gallery',
  function() {
    get_id(true);
    add_filter('wp_get_attachment_link','replace');
  }
);

The correct answer is probably to use the post_gallery hook to hijack the rest of the gallery shortcode, or just overwrite the gallery shortcode with one of your own.