Thanks Vancoder for your help!!! I have been ripping my hair out all day trying to figure it out. Funny enough your solution was one of half a dozen I wrote on my whiteboard, BUT I had no clue how to actually create an array with that as a key. I am a noob like that. Below is the code I ended up using with some comments for anyone else trying to do this!
<?php
$terms = get_terms("crb_issues");
$issue_archive = array( ); // creates an array for all terms inside of crb_issues tax using the custom field "issue_date" as the key
foreach ( $terms as $term ) {
$issue_date = get_field( 'issue_date', $term );
$issue_archive[$issue_date] = $term->name;
if (++$i == 4) break; //Stops the foreach after 4
}
krsort( $issue_archive, SORT_NUMERIC ); //sorts the issue_archive array from high to low
foreach ( $issue_archive as $issue_date => $term_name ) {
echo "<li>" . $term_name . " " . $issue_date . "</li>"; //displays the term name and customfield issue_date
}
?>
If anyone has better comments for this code please let me know, or better ways to handle anything. But I think this is a pretty awesome solution.