Properly loop through and paginate page on each function call

Question

Alright, I need some help and I don't know if I'm doing it the correct way, but I wanted to see if the community can guide me in a better direction.

Here is what's going on:

1. I have a register_rest_route() that calls the function (Not included in this code).

2. I make a call to grab the endpoint headers where it contains the total pages $headers['headers']['x-wp-totalpages'] which is 80.

3. I then loop through all the pages and call wp_remote_get to grab 25 posts from each page.

4. Next, I look through each post of the 25 posts and create posts on the new website.

Some details:

I have 795 posts on the website that contains all the data, when I run the current function it loops through all 795 posts and actually imports them, so it semi works, but there are bugs.

---

What I'm trying to achieve:

Does anyone know how I can format this function so that my wp_remote_get() $page param get incremented only when I hit the endpoint up until $headers['headers']['x-wp-totalpages'] and then reset back to $page=1 and not use a for loop that automatically loops through all 80 pages on a single request?

---

So I have the following function:

public function get_posts_via_rest_api(): void
    {
        // Pull in post_exists method for endpoints.
        if (!is_admin()) {
            require_once(ABSPATH . 'wp-admin/includes/post.php');
        }

        // Get the REST API Headers
        $headers = wp_remote_head('https://website-im-getting-data-from.com/wp-json/wp/v2/posts');

        // Start at page 1 and loop through all 80 pages from x-wp-totalpages
        for ($i = 1; $i <= $headers['headers']['x-wp-totalpages']; $i++) {
            $response = wp_remote_get(
                add_query_arg( [
                    'page' => $i,
                    'per_page' => 25,
                    'post_status' => 'publish',
                ], 'https://website-im-getting-data-from.com/wp-json/wp/v2/posts')
            );

            // Successful response? Proceed.
            if (wp_remote_retrieve_response_code($response) === 200) {
                try {
                    $posts = json_decode(
                        wp_remote_retrieve_body($response),
                        false,
                        512,
                        JSON_THROW_ON_ERROR
                    );
                } catch (Exception) {
                    return;
                }

                // Loop through the 25 posts from the identified page
                foreach ($posts as $post) {
                    $current_post_id = post_exists($post->title->rendered);
                    if ($current_post_id === 0) {
                        $my_post = [
                            'post_type'     => 'post',
                            'post_status'   => 'pending',
                            'post_title'    => wp_strip_all_tags($post->title->rendered),
                            'post_content'  => wp_strip_all_tags($post->content->rendered),
                            'post_excerpt'  => wp_strip_all_tags($post->excerpt->rendered),
                            'post_author'   => 1,
                            'post_date'     => $post->date,
                        ];

                        // Insert post.
                        $post_id = wp_insert_post($my_post);
                        wp_set_object_terms($post_id, 'Global', 'category');
                        wp_set_object_terms($post_id, 'Global', 'post_tag');
                        echo "ID: " . $post->id . " - Title: " . $post->title->rendered . " has been imported.\n";
                    }
                }
            }
        }
    }

Answer 1

Instead of fetching the first page, then looping over the total pages count from a variable, instead store it as an option, and use a second value to specify next page.

This way, you retrieve the next page number, fetch it, process the posts, then update the total pages options using the newer header from that request, and add 1 to the next page option and save it. If next page is larger than total pages reset it to 1.