Thanks Vancoder for your help!!! I have been ripping my hair out all day trying to figure it out. Funny enough your solution was one of half a dozen I wrote on my whiteboard, BUT I had no clue how to actually create an array with that as a key. I am a noob like that. Below is the code I ended up using with some comments for anyone else trying to do this!

    <?php
    $terms = get_terms("crb_issues");
    $issue_archive = array( ); // creates an array for all terms inside of crb_issues tax using the custom field "issue_date" as the key
    foreach ( $terms as $term ) {
    	$issue_date = get_field( 'issue_date', $term );
    	$issue_archive[$issue_date] = $term->name;
    }
    
    krsort( $issue_archive, SORT_NUMERIC ); //sorts the issue_archive array from high to low
    
    foreach ( $issue_archive as $issue_date => $term_name ) {
    	echo "<li>" . $term_name . " " . $issue_date . "</li>"; //displays the term name and customfield issue_date
    	if (++$i == 4) break; //Stops the foreach after 4
    }
    ?>

If anyone has better comments for this code please let me know, or better ways to handle anything. But I think this is a pretty awesome solution.