on attachment.php, how to display previous and next attachment links that follow the same order as a custom WP Query

Question

Current behavior

I have a media page that uses a custom WP_Query to return all thumbnails of media library attachments (regardless of post parent) with a custom taxonomy of media_visibility that has the term of published. This is working as expected on the media page. It's using my template file page-media.php. Relevant code sample:

  // how many media thumbnails to show at first.
  $thumbnails_to_show = 42;

  //Protect against arbitrary paged values
  $paged = ( get_query_var( 'paged' ) ) ? absint( get_query_var( 'paged' ) ) : 1;

        // query media
        $media_query_args = array(
            'post_type'   => 'attachment',
            'post_status' => 'any',
            'order' => 'DESC',
            'orderby' => 'date',
            'paged' => $paged,
            'tax_query' => array(
                    'relation' => 'AND',
                    array(
                        'taxonomy' => 'media_visibility',
                        'field' => 'slug',
                        'terms' => 'published',
                    ),
            ),
            'posts_per_page' => $thumbnails_to_show,
        );
        $media_query = new WP_Query ($media_query_args);

        $thumbnails = array();

        if ( $media_query->have_posts() ) : while ( $media_query->have_posts() ) : $media_query->the_post();
                // store thumbnails in array
                $thumbnails[] = wp_get_attachment_link( get_the_ID(), 'thumbnail', true );
            endwhile;

        endif; // end of media loop

When one clicks on a link on a thumbnail image, it loads a page showing the attachment and previous / next links using my attachment.php file. Relevant code sample:

    while ( have_posts() ) :
        the_post();

        get_template_part( 'template-parts/content', get_post_type() );

        previous_post_link();
        next_post_link();

        if ( comments_open() || get_comments_number() ) :
            comments_template();
        endif;

    endwhile; // End of the loop.

Here, there seems to be an issue with the order of posts / the different loops. On some (maybe all) attachment pages, previous_post_link() is displaying a link to the same attachment shown rather than the previous one, and next_post_link() is not displaying anything.

I also have another sort of page (a custom post type called "year story") that has a slightly different WP_Query to display multiple thumbnails - it's restricting the attachments to those that have the same terms in a couple of taxonomies. Full code in single-year_story.php:

        //get taxonomies from the post
        $this_ensemble = wp_get_post_terms(get_the_ID(),'ensemble')[0]->slug;
        $this_year = wp_get_post_terms(get_the_ID(),'vhs_year')[0]->slug;

        // how many media thumbnails to show at first
        $thumbnails_to_show = 6;

        // query media
        $media_query_args = array(
            'post_type'   => 'attachment',
            'post_status' => 'any',

            'tax_query' => array(
                    'relation' => 'AND',
                    array(
                        'taxonomy' => 'ensemble',
                        'field' => 'slug',
                        'terms' => $this_ensemble,
                    ),
                    array(
                        'taxonomy' => 'vhs_year',
                        'field' => 'slug',
                        'terms' => $this_year,
                    ),
                    array(
                        'taxonomy' => 'media_visibility',
                        'field' => 'slug',
                        'terms' => 'published',
                    ),
            ),

            'posts_per_page' => $thumbnails_to_show,
        );
        $media_query = new WP_Query ($media_query_args);

        $thumbnails = array();

        if ( $media_query->have_posts() ) : while ( $media_query->have_posts() ) : $media_query->the_post();
                // store thumbnails in array
                $thumbnails[] = wp_get_attachment_link( get_the_ID(), 'thumbnail', true );

            endwhile;

        endif; // end of media loop

Note that in both of these cases, the attachments don't necessarily have post parents. I set up the site to be fairly dynamic with visitors able to upload their own attachments, and I don't want to assign post parents to them when this happens (because we might not have created the corresponding year story post yet).

Desired behavior

What I want on the attachment page is to have previous and next text links that page through the attachment pages in the same order they appear on the WP_Query on the referring page (like a year story or media page) and only including those attachments shown on the referring page's WP_Query. If someone comes directly to an attachment page, I want these previous and next links to behave the same way as if someone came from the media page. If someone comes to the attachment page from an attachment in the body of a post or page, I want to either:

1. have the previous and next links cycle through the attachments with the same post parent, or
2. hide the previous and next links - they're not very important for this case.

Other attempts

I tried some other approaches like the_post_navigation() and get_previous_image_link() / get_next_image_link() but did not get what I was trying to achieve here. I also tried replacing the loop on attachment.php with the same custom WP_Query that I used on page-media.php but this ended up always displaying the first attachment returned in the query, rather than the attachment specified in the URL.

Links

View the media page on staging site

View a year story page on staging site

(In order to view the above links, log in as username: stackexchange password: guest)

Theme code

This customized theme is based on underscores.