Thanks to `Tom J Nowell` I found the solution. It was so simple that I felt a little shame not being able to figure it out myself. 

First we need a function to 'insert' our custom taxonomy custom field to our WP Object. In this case the CPT is 'tour_category' and the Custom Field is 'order_number'

    function terms()
        {
          return array_map(function($term) {
              $term->order_number = get_field('order_number', $term);
              return $term;
            }, get_terms([
              'taxonomy' => 'tour_category',
              'orderby'=> 'order_number',
              'order' => 'DESC'
          ]));
        }

then, we use it in our loop:

    $my= terms();
    $num = array_column($my, 'order_number');
    array_multisort($num, SORT_ASC, $my);

    foreach($my as $term) { .... }