Manually setting `is_404 = false;` fixed my issue. However I'm not sure this is the best way to do it. I tried using the `pre_get_posts` filter instead without any luck.

Anyway, for anyone else in the same boat, you can do this to get rid of the 404 state:
 
    public function add_response_template($template) {
      global $wp_query;
      if ( 'my_custom_url' === get_query_var( 'pagename' ) ) {
        $wp_query->is_404 = false;
        $template = trailingslashit( dirname( __FILE__ ) ) . 'templates/custom-page-template.php';
      }

      return $template;
    }


And to update the document title (The stuff inside `<title>` in the `<head>` section) here's a snippet for making that work nicely too.

    add_filter( 'document_title_parts', function($title_arr) {
      if ( 'my_custom_url' === get_query_var('pagename') ) {
        $title_arr['title'] = "Document title for my custom route";
      }

      return $title_arr;
    }, 10, 1 );

If anyone knows of a better way please let me know.