add_rewrite_tag broke permalinks that doesn't use that specific tag

Question

I have a custom post type that I'd like to have a permalink slug based on its taxonomy. All the posts of this CPT have one, and only one, term always marked on this specific taxonomy.
That's my code:

function plugin_domain_register_post_type(){

    add_rewrite_tag('%event_segment%', '([^&]+)');

    register_post_type( 'event',
        array(
            'public' => true,
            'rewrite' => array( 'slug' => '%event_segment%'),
            'has_archive' => false,
            'hierarchical' => false,
            'supports' => array( 'title', 'thumbnail')
        )
    );

}
add_action( 'init', 'plugin_domain_register_post_type' );

function plugin_domain_permalinks($post_link, $post) {
    if (is_object($post) && $post->post_type === 'event') {
        $terms = wp_get_object_terms($post->ID, 'segments');
        if ($terms) {
            return str_replace('%event_segment%' , $terms[0]->slug, $post_link);
        }
    }
    return $post_link;
}
add_filter('post_type_link', 'plugin_domain_permalinks', 10, 3);

Okay, so it actually works, the permalinks changed and it goes to the right post. However, all the posts and pages (everything that's not of this CPT) that do not have the %event_segment% tag on the slug will literally just display the home page.

I've noticed this happens because of the add_rewrite_tag('%event_segment%', '([^&]+)'); part. The filter doesn't seem to be the issue, since even without the rewrite tag, it works as intended, and modifies the permalinks.

I don't know if this is relevant, but I'm on a Varying Vagrant Vagrants development environment, and it uses nginx, not apache.

What's going on here? Is there something I did wrong or forgot to do?