Edit: If anyone runs across a similar issue; the problem was I had Cloudflare DNS Proxy turned on for my server and that was causing the audio issues. It seems to be a semi-common problem from Googling it.
I uploaded my audio files to BunnyCDN storage and then turned CloudFlare DNS Proxy back on. Everything works fine now. I'm removing links in this post for privacy reasons.
My site is running on Vultr/RunCloud/Cloudflare. I've done a lot of testing; turned off all plugins, disabled all cache/CDN, disabled Cloudflare cache, disabled Redis Object Cache, changed to default theme and nothing fixed it. The weird thing is that it's totally random, sometimes the audio player will work fine, other times it won't.
Locally hosted mp3/ogg files on my server will not seek, when you try to scrub through the audio it resets back to the beginning. This is happening in Chrome and Edge desktop and mobile, works fine in Firefox. Also, on iPad/iPhone the scrub bar doesn't even react to any touches.
So I uploaded the same mp3/ogg files to my BunnyCDN Storage bucket and those audio files work perfectly across all browsers and iPad/iPhone.
Here is a test page: https://www.testsite.com/audiotestpage/
(you might have to F5 a few times to see the issue)
If you take this file and put it directly in Chrome, try to scrub through the audio and it'll reset back to the start: https://www.testsite.com/testfile.mp3
Now here's the same file hosted on BunnyCDN and do the same and it works fine: https://testsite.net/testfile.mp3
I'm getting no errors in Chrome Dev. I'm guessing this a server side issue but I'm totally lost on what it could be.