How get JSON from external API on every page load if not existing in transient?

Question

Clarification:

1. I'm building a site that will show the profiles of different agents (or specialists) in grid based cards. Each of these profile cards will show the online/offline status of the agent and visitors should be able to click a phone link (on mobile phones) to dial the phone number to respective agent.

2. In order to house the data connected to each agent, I've created a custom post type (cpt) "agent". This cpt have several custom fields that is connected to the WP REST api. These will house the individual fetched data objects of each agent.

3. The source of the agent data comes from an external company via their api. They use basic authentication (user:pass).

4. My goal is to fetch the data from that api, and save as a new "agent" (the cpt) or update already existing posts in case they differ from the source. Each part of this data (such as id, agentCode, phone number and so on) should be fetched and saved into the corresponding custom fields.

To summarize:

Disclaimer: I don't have a clue what I'm doing (really). I'm a designer, not a developer, but you can't blame me for not trying ;).

• What is the best option to fetch the different online statuses automatically "on-the-fly" in real time?
• What is the most efficient way of fetching (external) data like these? Optimally everything happens "on-the-fly" as soon as the source differ from the site. At least it should run a check every page update. -What about caching? I have read a few lines about transients. Would that work in this context?
• Obviously I would absolutely LOVE code suggestions covering everything above.

The code down below works when performing a manual request in the browser (note that actual urls are replaced with dummy ones). However I need it to actually work on its own, automatically but haven't yet succeeded. I tried wrapping it all inside an action hook (after_setup_theme) but it didn't work.

php

  add_action('wp_ajax_nopriv_get_all_agents_from_api', 'get_all_agents_from_api');
  add_action('wp_ajax_get_all_agents_from_api', 'get_all_agents_from_api');
  add_action('wp_ajax_nopriv_update_agents_on_api', 'update_agents_on_api');
  add_action('wp_ajax_update_agents_on_api', 'update_agents_on_api');

function get_all_agents_from_api() {

    $file = get_stylesheet_directory() . '/agentData.txt';
    $profiles = [];

    $username = '[email protected]';
    $password = 'tJ75F9tG&QISC#$2t';

    $args = array(
        'headers' => array(
            'Authorization' => 'Basic ' . base64_encode( $username . ':' . $password )
        )
    );
    // Get a list of our local agents to avoid duplicates
    $local = wp_remote_retrieve_body(wp_remote_get(get_site_url(null, '/wp-json/wp/v2/agents')));
    $local = json_decode($local, true);
    $localagents=Array();
    foreach($local as $agent)
    {
      $localagents[]=$agent['slug']; // Not the best ID, but good enough for now
    }

    $results = wp_remote_retrieve_body(wp_remote_get('https://api.somefancyurl.com/mockupagents/v1/Agents', $args));
    file_put_contents($file, "Current Page: 1\n\n", FILE_APPEND);

    // JSON decode the response
    $results = json_decode($results);

    if(!is_array($results) || empty($results)) {
        return false;
    }

    foreach($results as $agent)
    {
        foreach($agent->profiles as $profile) {

            $profile_slug = sanitize_title($profile->profileName . '-' . $profile->profileId);
            if(in_array($profile_slug, $localagents)){continue;} // Already got it

            $inserted_profile = wp_insert_post([
                'post_name' => $profile_slug,
                'post_title' => $profile->profileName,
                'post_content' => '',
                'post_type' => 'agent',
                'post_status' => 'publish'
            ], $wp_error=true);

            if(is_wp_error($inserted_profile)) {
                continue;
            }

            // Added to profile_data: agentId, agent_status_available, agent_status_inCall and agent_status_loggedOut.
            $profile_data = [
                'field_6295da4e335b1' => 'profileId',
                'field_6297663deba92' => 'agentId',
                'field_62447e691eb7f' => 'profileName',
                'field_62447e8e1eb80' => 'serviceName',
                'field_629617a81de38' => 'accessNumber',
                'field_624c212c894e0' => 'serviceId',
                'field_62c2f62c8e9b2' => 'agent_status_available',
                'field_62c2f6848e9b3' => 'agent_status_inCall',
                'field_62c2f6b58e9b4' => 'agent_status_loggedOut'
            ];

            foreach($profile_data as $key => $profileName) {
                update_field($key, $profile->$profileName, $inserted_profile);
            }
            update_field('field_6245952692451', $agent->notes, $inserted_profile);

        }

    }

}

function idtrim($url)
{
  $slash=strrpos($url, '/');
  return substr($url, $slash+1);
}

function update_agents_on_api() {
  // Set up RPC auth
  $username = '[email protected]';
  $password = 'tJ75F9tG&QISC#$2t';

  $patchargs = array(
    'method' => 'PATCH',
    'headers' => array(
      'Authorization'=>'Basic '.base64_encode($username.':'.$password),
      'Content-type'=>'application/json'
    )
  );
  $args = array(
    'headers' => array(
      'Authorization'=>'Basic '.base64_encode($username.':'.$password),
      'Content-type'=>'application/json'
    )
  );
  // Get our current list of remote agents to check if we should create or update
  $results=wp_remote_retrieve_body(wp_remote_get('https://api.somefancyurl.com/mockupagents/v1/Agents', $args));
  $results=json_decode($results, true);
  $remoteagents=Array();
  foreach($results as $agent)
  {
    if(!isset($agent['profiles'][0])){continue;} // No profile=no name to turn into a slug
    $profile=$agent['profiles'][0];
    $slug=sanitize_title($profile['profileName']);//.'-'.$profile['profileId']);
    $remoteagents[$slug]=Array($agent['agentId'], $agent['profiles'][0]['profileId']);
  }
  // Get a list of our local agents
  $local=wp_remote_retrieve_body(wp_remote_get(get_site_url(null, '/wp-json/wp/v2/agents')));
  $local=json_decode($local, true);
  $localagents=Array();
  foreach($local as $agent)
  {
    $names=explode(' ', $agent['title']['rendered']);
    $agentinfo=Array(
      'agentName'=>$agent['title']['rendered'],
      'notes'=>htmlentities($agent['acf']['about_advisor']),
      'agentNumber'=>$agent['id'],
      'phoneNumber'=>'37907107795',
      'pin'=>$agent['id'],
      );
    $profileinfo=Array();
    if(!isset($remoteagents[$agent['slug']]))
    { // Agent doesn't exist yet, create it
//print('Creating '.$agent['title']['rendered'].' because $remoteagents['.$agent['slug'].'] is undefined<br />');
      $profileinfo['profileName']=$agent['slug'];//htmlentities($agent['acf']['screen_name']);
      $profileinfo['profileNumber']=random_int(1,999);
      $profileinfo['serviceId']=427;//random_int(1,999);
      $args['body']=json_encode($agentinfo);
      $r=wp_remote_post('https://api.somefancyurl.com/mockupagents/v1/Agents', $args);
      if(substr($r['response']['code'],0,1)!='2' && substr($r['response']['code'],0,1)!='3'){print(1);break;}
      // Profile
      $profileinfo['agentId']=idtrim($r['headers']['location']);
      if(!$profileinfo['agentId']){continue;} // Invalid
      $args['body']=json_encode($profileinfo);
      $r=wp_remote_post('https://api.somefancyurl.com/mockupagents/v1/Profiles', $args);
      if(substr($r['response']['code'],0,1)!='2' && substr($r['response']['code'],0,1)!='3'){print(2);break;}
    }else{
      // Update agent
      $id=$remoteagents[$agent['slug']][0];
      $profileinfo['agentId']=$id;
      $patchargs['method']='PUT';
      $patchargs['body']=json_encode($agentinfo);
      $r=wp_remote_request('https://api.somefancyurl.com/mockupagents/v1/Agents'.$id, $patchargs);
      if(substr($r['response']['code'],0,1)!='2' && substr($r['response']['code'],0,1)!='3'){print(3);break;}
      // Profile
      $id=$remoteagents[$agent['slug']][1];
      $patchargs['method']='PATCH';
      $patchargs['body']=json_encode($profileinfo);
      $r=wp_remote_request('https://api.somefancyurl.com/mockupagents/v1/Profiles/'.$id, $patchargs);
      if(substr($r['response']['code'],0,1)!='2' && substr($r['response']['code'],0,1)!='3'){print(4);break;}
    }
  }
}

If anyone could please help me solve this I will be forever grateful! //Seb

Answer 1

What is the best option to fetch the different online statuses automatically "on-the-fly" in real time?

What is the most efficient way of fetching (external) data like these? Optimally everything happens "on-the-fly" as soon as the source differ from the site. At least it should run a check every page update. -What about caching? I have read a few lines about transients. Would that work in this context?

Not on the fly.

Right now you've ran head first into a fundamental trade off. The more "realtime" you make this, the slower it will be.

• At an absolute minimum your page will always be as slow, or slower, than what the API is.
• If I refresh the page half a second later it'll pay that cost all over again even though it's just fetched the data.
• If that service goes down, so does that page

Your tradeoff is latency vs speed. You can make the page blazing fast/cheap to generate at the cost of immediacy, or, you can make it fully realtime but that costs resources/time/speed.

Opting to always go for max 100% speed doesn't work, not just because it slows your page down, but because by the time the page has loaded and the viewer has looked through the data it's already out of date by your criteria.

So instead:

• Don't request on every page load for every person. You never asked for bespoke couture responses tailored ot each viewer, you asked for up to date responses. Why make the request multiple times at the same time for multiple viewers?
• Satisfice, pull in the data in the background in a cron job and update your posts that way, say every 5 minutes. Put that 5 minute figure in the page so users know, and it'll be clear that the data is never older than 5 minutes. This also reduces your API calls to the remote server which might save cost and bandwidth. Or perhaps more often than that, the interval is up to you.
  • you want to put the fetching of the data and the display of the data as far away from eachother as possible, ideally in separate requests so that they can happen at the same time. You don't want to involve the backoffice everytime a customer arrives, just send regular updates to front of house with the "new information". Displaying the data you have stored is cheap, fast, and scales. Updating that data is not.
• always cache remote data, even a tiny cache expiration of 10 seconds will have enormous benefits if you have a lot of visitors, and almost no difference if one or two people visit. Just make sure this cache lasts longer than it takes to make the requests.
• you may be able to conceal the almost realtime/pseudo realtime nature of things with smart UX, e.g. most services don't tell you that XYZ is online, rather they tell you they were recently online, or consider any presence in the last 5 minutes to be online. If you can say "seen 10 minutes ago" then that's enough for most people.

If you need realtime status on wether they're online or not, then you may never be able to get that. For true realtime status you need your frontend browser connected via a socket the remote API so that the 3rd party can push that data to your viewers. Otherwise you'll need to do what almost all sites do but pretend they don't: "Ask really really often and pretend it's realtime", aka polling via AJAX.

And store your data in your agent CPT.

I'd recommend reading up on WP Cron for scheduling actions:

• https://developer.wordpress.org/plugins/cron/
• https://developer.wordpress.org/plugins/cron/understanding-wp-cron-scheduling/