There is no way to make the WP Cron system work without the website being hit. If you would like to prevent the user from having to be the one to initiate the cache refresh you can set up an actual scheduled task with something like AWS Lambda or CronJob.org to hit the page every minute.
However, even if the scheduled task runs every minute, there is still a small window of close-to-a-minute where a user might still be the one to initiate the WP Cron.
A better solution, if your cache system has the proper hooks, is to set up the scheduled task to hit a url with a query string something like
?cache_refresh=true and then parse that to refresh the cache (or expire it and set it again) so that a user will never be the one to initiate the refresh. With this method, you would set your expiration time a lot higher (like a day) and then make the scheduled task run every two hours to give the scheduled task full control over when the cache is refreshed. In doing so, you would no longer need to use the WP Cron system at all.
If your plugin does not have the hooks to do this, you may want to check to see if it is using Wordpress Transients to achieve the caching. If it is, you may be able to use the Transient API to force the refresh - or just scrap the plugin all together and use the Transient API directly with the scheduled tasks to have better control over the process.
EDIT: Additionally, maybe a bit overkill for this application, you could make your WP Cron activate an asynchronous task that will do the work in the background - most easily done with a library such as this one.