Thanks to Tom J Nowell I found the solution. It was so simple that I felt a little shame not being able to figure it out myself.

First we need a function to 'insert' our custom taxonomy custom field to our WP Object. In this case the CPT is 'tour_category' and the Custom Field is 'order_number'

function terms()

{
  return array_map(function($term) {
      $term->order_number = get_field('order_number', $term);
      return $term;
    }, get_terms([
      'taxonomy' => 'tour_category',
      'orderby'=> 'order_number',
      'order' => 'DESC'
  ]));
}

then, we use it in our loop:

$my= terms();
$num = array_column($my, 'order_number');
array_multisort($num, SORT_ASC, $my);

foreach($my as $term) { .... }   

Thanks to Tom J Nowell I found the solution. It was so simple that I felt a little shame not being able to figure it out myself.

First we need a function to 'insert' our custom taxonomy custom field to our WP Object. In this case the CPT is 'tour_category' and the Custom Field is 'order_number'

function terms()
    {
      return array_map(function($term) {
          $term->order_number = get_field('order_number', $term);
          return $term;
        }, get_terms([
          'taxonomy' => 'tour_category',
          'orderby'=> 'order_number',
          'order' => 'DESC'
      ]));
    }

then, we use it in our loop:

$my= terms();
$num = array_column($my, 'order_number');
array_multisort($num, SORT_ASC, $my);

foreach($my as $term) { .... }