Alright, I need some help and I don't know if I'm doing it the correct way, but I wanted to see if the community can guide me in a better direction.

Here is what's going on:

  1. I have a register_rest_route() that calls the function (Not included in this code).

  2. I make a call to grab the endpoint headers where it contains the total pages $headers['headers']['x-wp-totalpages'] which is 80.

  3. I then loop through all the pages and call wp_remote_get to grab 25 posts from each page.

  4. Next, I look through each post of the 25 posts and create posts on the new website.

Some details:

I have 795 posts on the website that contains all the data, when I run the current function it loops through all 795 posts and actually imports them, so it semi works, but there are bugs.

What I'm trying to achieve:

Does anyone know how I can format this function so that my wp_remote_get() $page param get incremented only when I hit the endpoint up until $headers['headers']['x-wp-totalpages'] and then reset back to $page=1 and not use a for loop that automatically loops through all 80 pages on a single request?

So I have the following function:

public function get_posts_via_rest_api(): void
        // Pull in post_exists method for endpoints.
        if (!is_admin()) {
            require_once(ABSPATH . 'wp-admin/includes/post.php');

        // Get the REST API Headers
        $headers = wp_remote_head('https://website-im-getting-data-from.com/wp-json/wp/v2/posts');

        // Start at page 1 and loop through all 80 pages from x-wp-totalpages
        for ($i = 1; $i <= $headers['headers']['x-wp-totalpages']; $i++) {
            $response = wp_remote_get(
                add_query_arg( [
                    'page' => $i,
                    'per_page' => 25,
                    'post_status' => 'publish',
                ], 'https://website-im-getting-data-from.com/wp-json/wp/v2/posts')

            // Successful response? Proceed.
            if (wp_remote_retrieve_response_code($response) === 200) {
                try {
                    $posts = json_decode(
                } catch (Exception) {

                // Loop through the 25 posts from the identified page
                foreach ($posts as $post) {
                    $current_post_id = post_exists($post->title->rendered);
                    if ($current_post_id === 0) {
                        $my_post = [
                            'post_type'     => 'post',
                            'post_status'   => 'pending',
                            'post_title'    => wp_strip_all_tags($post->title->rendered),
                            'post_content'  => wp_strip_all_tags($post->content->rendered),
                            'post_excerpt'  => wp_strip_all_tags($post->excerpt->rendered),
                            'post_author'   => 1,
                            'post_date'     => $post->date,

                        // Insert post.
                        $post_id = wp_insert_post($my_post);
                        wp_set_object_terms($post_id, 'Global', 'category');
                        wp_set_object_terms($post_id, 'Global', 'post_tag');
                        echo "ID: " . $post->id . " - Title: " . $post->title->rendered . " has been imported.\n";

1 Answer 1


Instead of fetching the first page, then looping over the total pages count from a variable, instead store it as an option, and use a second value to specify next page.

This way, you retrieve the next page number, fetch it, process the posts, then update the total pages options using the newer header from that request, and add 1 to the next page option and save it. If next page is larger than total pages reset it to 1.

  • Also note, that just because you fetched the posts, doesn't mean you have to also create them on the same request. You could just as easily check the posts to see if they exist, then schedule a cron job to retrieve that individual post in the near future. This way you can loop through the pages much faster and request more posts per page, e.g. 50, without needing to create 50 posts in that request ( they can happen on smaller future cron jobs that create a single post )
    – Tom J Nowell
    Mar 20, 2022 at 22:49
  • as an aside, if you had access to WP CLI, this could all be done in one big job as a single terminal command
    – Tom J Nowell
    Mar 20, 2022 at 22:49
  • I don't utilize the CLI as this will be a custom solution, but I like your idea about fetching the posts and creating them later, but I guess I'm having a hard time visualizing the crons position in retrieving each individual post if the first time it runs, there will be like 767 posts, but thanks for the explanation on some of the other items! They're very useful. The thing now is that it will hit each individual page of 80 and who knows when it would hit the page with the post that needs to be imported.
    – Test2
    Mar 21, 2022 at 0:34
  • then there will be 767 new cron jobs, note that these are not the same cron job that fetches the pages, this would be a new cron job that retrieves just that post and only that post, then uses wp_insert_post. You can pass the ID via the cron scheduling so it's passed as a parameter. As for who knows when it would hit the page with a post that needs importing, you would need to request every page to find that out
    – Tom J Nowell
    Mar 21, 2022 at 1:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.