0

Clarification:

  1. I'm building a site that will show the profiles of different agents (or specialists) in grid based cards. Each of these profile cards will show the online/offline status of the agent and visitors should be able to click a phone link (on mobile phones) to dial the phone number to respective agent.

  2. In order to house the data connected to each agent, I've created a custom post type (cpt) "agent". This cpt have several custom fields that is connected to the WP REST api. These will house the individual fetched data objects of each agent.

  3. The source of the agent data comes from an external company via their api. They use basic authentication (user:pass).

  4. My goal is to fetch the data from that api, and save as a new "agent" (the cpt) or update already existing posts in case they differ from the source. Each part of this data (such as id, agentCode, phone number and so on) should be fetched and saved into the corresponding custom fields.

To summarize:

Disclaimer: I don't have a clue what I'm doing (really). I'm a designer, not a developer, but you can't blame me for not trying ;).

  • What is the best option to fetch the different online statuses automatically "on-the-fly" in real time?
  • What is the most efficient way of fetching (external) data like these? Optimally everything happens "on-the-fly" as soon as the source differ from the site. At least it should run a check every page update. -What about caching? I have read a few lines about transients. Would that work in this context?
  • Obviously I would absolutely LOVE code suggestions covering everything above.

The code down below works when performing a manual request in the browser (note that actual urls are replaced with dummy ones). However I need it to actually work on its own, automatically but haven't yet succeeded. I tried wrapping it all inside an action hook (after_setup_theme) but it didn't work.

php

  add_action('wp_ajax_nopriv_get_all_agents_from_api', 'get_all_agents_from_api');
  add_action('wp_ajax_get_all_agents_from_api', 'get_all_agents_from_api');
  add_action('wp_ajax_nopriv_update_agents_on_api', 'update_agents_on_api');
  add_action('wp_ajax_update_agents_on_api', 'update_agents_on_api');

function get_all_agents_from_api() {

    $file = get_stylesheet_directory() . '/agentData.txt';
    $profiles = [];

    $username = 'xxxxx.xxxxxxx@xxxxxxxx.com';
    $password = 'tJ75F9tG&QISC#$2t';

    $args = array(
        'headers' => array(
            'Authorization' => 'Basic ' . base64_encode( $username . ':' . $password )
        )
    );
    // Get a list of our local agents to avoid duplicates
    $local = wp_remote_retrieve_body(wp_remote_get(get_site_url(null, '/wp-json/wp/v2/agents')));
    $local = json_decode($local, true);
    $localagents=Array();
    foreach($local as $agent)
    {
      $localagents[]=$agent['slug']; // Not the best ID, but good enough for now
    }

    $results = wp_remote_retrieve_body(wp_remote_get('https://api.somefancyurl.com/mockupagents/v1/Agents', $args));
    file_put_contents($file, "Current Page: 1\n\n", FILE_APPEND);

    // JSON decode the response
    $results = json_decode($results);

    if(!is_array($results) || empty($results)) {
        return false;
    }

    foreach($results as $agent)
    {
        foreach($agent->profiles as $profile) {

            $profile_slug = sanitize_title($profile->profileName . '-' . $profile->profileId);
            if(in_array($profile_slug, $localagents)){continue;} // Already got it

            $inserted_profile = wp_insert_post([
                'post_name' => $profile_slug,
                'post_title' => $profile->profileName,
                'post_content' => '',
                'post_type' => 'agent',
                'post_status' => 'publish'
            ], $wp_error=true);

            if(is_wp_error($inserted_profile)) {
                continue;
            }

            // Added to profile_data: agentId, agent_status_available, agent_status_inCall and agent_status_loggedOut.
            $profile_data = [
                'field_6295da4e335b1' => 'profileId',
                'field_6297663deba92' => 'agentId',
                'field_62447e691eb7f' => 'profileName',
                'field_62447e8e1eb80' => 'serviceName',
                'field_629617a81de38' => 'accessNumber',
                'field_624c212c894e0' => 'serviceId',
                'field_62c2f62c8e9b2' => 'agent_status_available',
                'field_62c2f6848e9b3' => 'agent_status_inCall',
                'field_62c2f6b58e9b4' => 'agent_status_loggedOut'
            ];

            foreach($profile_data as $key => $profileName) {
                update_field($key, $profile->$profileName, $inserted_profile);
            }
            update_field('field_6245952692451', $agent->notes, $inserted_profile);

        }

    }

}

function idtrim($url)
{
  $slash=strrpos($url, '/');
  return substr($url, $slash+1);
}

function update_agents_on_api() {
  // Set up RPC auth
  $username = 'xxxxx.xxxxxxx@xxxxxxxx.com';
  $password = 'tJ75F9tG&QISC#$2t';

  $patchargs = array(
    'method' => 'PATCH',
    'headers' => array(
      'Authorization'=>'Basic '.base64_encode($username.':'.$password),
      'Content-type'=>'application/json'
    )
  );
  $args = array(
    'headers' => array(
      'Authorization'=>'Basic '.base64_encode($username.':'.$password),
      'Content-type'=>'application/json'
    )
  );
  // Get our current list of remote agents to check if we should create or update
  $results=wp_remote_retrieve_body(wp_remote_get('https://api.somefancyurl.com/mockupagents/v1/Agents', $args));
  $results=json_decode($results, true);
  $remoteagents=Array();
  foreach($results as $agent)
  {
    if(!isset($agent['profiles'][0])){continue;} // No profile=no name to turn into a slug
    $profile=$agent['profiles'][0];
    $slug=sanitize_title($profile['profileName']);//.'-'.$profile['profileId']);
    $remoteagents[$slug]=Array($agent['agentId'], $agent['profiles'][0]['profileId']);
  }
  // Get a list of our local agents
  $local=wp_remote_retrieve_body(wp_remote_get(get_site_url(null, '/wp-json/wp/v2/agents')));
  $local=json_decode($local, true);
  $localagents=Array();
  foreach($local as $agent)
  {
    $names=explode(' ', $agent['title']['rendered']);
    $agentinfo=Array(
      'agentName'=>$agent['title']['rendered'],
      'notes'=>htmlentities($agent['acf']['about_advisor']),
      'agentNumber'=>$agent['id'],
      'phoneNumber'=>'37907107795',
      'pin'=>$agent['id'],
      );
    $profileinfo=Array();
    if(!isset($remoteagents[$agent['slug']]))
    { // Agent doesn't exist yet, create it
//print('Creating '.$agent['title']['rendered'].' because $remoteagents['.$agent['slug'].'] is undefined<br />');
      $profileinfo['profileName']=$agent['slug'];//htmlentities($agent['acf']['screen_name']);
      $profileinfo['profileNumber']=random_int(1,999);
      $profileinfo['serviceId']=427;//random_int(1,999);
      $args['body']=json_encode($agentinfo);
      $r=wp_remote_post('https://api.somefancyurl.com/mockupagents/v1/Agents', $args);
      if(substr($r['response']['code'],0,1)!='2' && substr($r['response']['code'],0,1)!='3'){print(1);break;}
      // Profile
      $profileinfo['agentId']=idtrim($r['headers']['location']);
      if(!$profileinfo['agentId']){continue;} // Invalid
      $args['body']=json_encode($profileinfo);
      $r=wp_remote_post('https://api.somefancyurl.com/mockupagents/v1/Profiles', $args);
      if(substr($r['response']['code'],0,1)!='2' && substr($r['response']['code'],0,1)!='3'){print(2);break;}
    }else{
      // Update agent
      $id=$remoteagents[$agent['slug']][0];
      $profileinfo['agentId']=$id;
      $patchargs['method']='PUT';
      $patchargs['body']=json_encode($agentinfo);
      $r=wp_remote_request('https://api.somefancyurl.com/mockupagents/v1/Agents'.$id, $patchargs);
      if(substr($r['response']['code'],0,1)!='2' && substr($r['response']['code'],0,1)!='3'){print(3);break;}
      // Profile
      $id=$remoteagents[$agent['slug']][1];
      $patchargs['method']='PATCH';
      $patchargs['body']=json_encode($profileinfo);
      $r=wp_remote_request('https://api.somefancyurl.com/mockupagents/v1/Profiles/'.$id, $patchargs);
      if(substr($r['response']['code'],0,1)!='2' && substr($r['response']['code'],0,1)!='3'){print(4);break;}
    }
  }
}

If anyone could please help me solve this I will be forever grateful! //Seb

10
  • 1
    I'm not 100% sure what it is you're trying to do, it seems like parts of your explanation are skipped over/assumed/missing. I'm also a little confused why you're using the old admin legacy AJAX if you're interacting with remote REST API endpoints. The best I can figure out is that you don't know how to run code in the background on a schedule, is that what you're asking how to do?
    – Tom J Nowell
    Aug 5 at 13:07
  • 1
    also $local = wp_remote_retrieve_body(wp_remote_get(get_site_url(null, '/wp-json/wp/v2/agents'))); is an extremely expensive way to get information from the same site, if you're already on the site that has the information running PHP code then making a HTTP request is the slowest most expensive way to get it. You can already just ask WP to run that request without using wp_remote_get, and you can just call the functions directly. It's like needing to remember something but rather than remembering it, you write down a request, send it it in the post to yourself then mail a reply back
    – Tom J Nowell
    Aug 5 at 13:09
  • you could see massive performance improvements by using rest_do_request when fetching the REST response of your local site instead. I'm also not seeing where update_field is coming from
    – Tom J Nowell
    Aug 5 at 13:10
  • Thank you @TomJNowell for your input. I've now tried to clarify my post as much as I can. I found the contents of get_all_agents_from_api() on some website. Had no idea it was that bad! Everything below is done by a colleague of mine.
    – Seblito
    Aug 5 at 20:20
  • if you can add a link to where you found that to your question it would provide a lot of useful context. I'd also change your password in case someone guesses the email account you used. As for the most efficient method, that would be fetching data in a cron job and storing it. Ideally when a user browses your site none of these API calls are happening as HTTP requests are slow, just have it show the data available at the moment on your site, and then update that data at regular intervals behind the scenes
    – Tom J Nowell
    Aug 7 at 11:41

1 Answer 1

1

What is the best option to fetch the different online statuses automatically "on-the-fly" in real time?

What is the most efficient way of fetching (external) data like these? Optimally everything happens "on-the-fly" as soon as the source differ from the site. At least it should run a check every page update. -What about caching? I have read a few lines about transients. Would that work in this context?

Not on the fly.

Right now you've ran head first into a fundamental trade off. The more "realtime" you make this, the slower it will be.

  • At an absolute minimum your page will always be as slow, or slower, than what the API is.
  • If I refresh the page half a second later it'll pay that cost all over again even though it's just fetched the data.
  • If that service goes down, so does that page

Your tradeoff is latency vs speed. You can make the page blazing fast/cheap to generate at the cost of immediacy, or, you can make it fully realtime but that costs resources/time/speed.

Opting to always go for max 100% speed doesn't work, not just because it slows your page down, but because by the time the page has loaded and the viewer has looked through the data it's already out of date by your criteria.

So instead:

  • Don't request on every page load for every person. You never asked for bespoke couture responses tailored ot each viewer, you asked for up to date responses. Why make the request multiple times at the same time for multiple viewers?
  • Satisfice, pull in the data in the background in a cron job and update your posts that way, say every 5 minutes. Put that 5 minute figure in the page so users know, and it'll be clear that the data is never older than 5 minutes. This also reduces your API calls to the remote server which might save cost and bandwidth. Or perhaps more often than that, the interval is up to you.
    • you want to put the fetching of the data and the display of the data as far away from eachother as possible, ideally in separate requests so that they can happen at the same time. You don't want to involve the backoffice everytime a customer arrives, just send regular updates to front of house with the "new information". Displaying the data you have stored is cheap, fast, and scales. Updating that data is not.
  • always cache remote data, even a tiny cache expiration of 10 seconds will have enormous benefits if you have a lot of visitors, and almost no difference if one or two people visit. Just make sure this cache lasts longer than it takes to make the requests.
  • you may be able to conceal the almost realtime/pseudo realtime nature of things with smart UX, e.g. most services don't tell you that XYZ is online, rather they tell you they were recently online, or consider any presence in the last 5 minutes to be online. If you can say "seen 10 minutes ago" then that's enough for most people.

If you need realtime status on wether they're online or not, then you may never be able to get that. For true realtime status you need your frontend browser connected via a socket the remote API so that the 3rd party can push that data to your viewers. Otherwise you'll need to do what almost all sites do but pretend they don't: "Ask really really often and pretend it's realtime", aka polling via AJAX.

And store your data in your agent CPT.

I'd recommend reading up on WP Cron for scheduling actions:

2
  • 1
    Thank you Tom for your time and effort!
    – Seblito
    yesterday
  • glad to be of help! Reading back on what I wrote a few days ago, I think the biggest take away is that you can make it look realtime without it actually being realtime, especially with a handful of minor UX changes and regular background polling. Try to separate the fetching of data from the API from the viewing of data too.
    – Tom J Nowell
    yesterday

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.