Correct Hook/Filter to amend category choices on post edit page

Question

I want to filter available terms for a user when they create/edit a post, I have assigned a custom field to the user where it returns an array of IDs for the categories they can edit.

In an action for load-post.php/load-post-new.php I call my function, where I can loop through the terms for the category taxonomy, and also retrieve my users available category IDs. I can then unset the term, which works when dumping the variable, however in the admin it appears the post category meta box is not affected by this, therefore I imagine I need to add a filter in order to get what I need. What would be the correct filter/hook to remove categories I do not need in the list?

Currently I am using:

add_action( 'load-post.php', array($this,'filter_user_categories'));
add_action( 'load-post-new.php', array($this,'filter_user_categories'));

Below is my functions inner code:

$current_user = get_current_user_id();
$user_field = "user_".$current_user;

$user_categories = get_field('post_categories', $user_field);

$terms = get_terms( array(
     'taxonomy' => 'category',
     'hide_empty' => false,
) );

$i=0;

foreach($terms as $term){

    if(!in_array($term->term_id,$user_categories)){
          echo $term->term_id;
          unset($terms[$i]);
    }
    $i++;
}

Answer 1

I managed to solve this issue using pre_get_termsas an action and then updating the query_vars

$current_user = get_current_user_id();
$user_field = "user_".$current_user;

$user_categories = get_field('post_categories', $user_field);

if(in_array('category',$terms->query_vars['taxonomy'])){
     $terms->query_vars['include'] = $user_categories;
}

return $terms;