I am using wp_remote_get() to retrieve different pages from the web. It is a news site.

Most of the time, this works successfully. However, sometimes, the request returns success (code 200), but the content of the page is differet from the page that is retrieved by a brwoser (the content of the page returned with this call is something along the lines of "page not found")

However if I use any browser to retrieve the page in question, the page is successfully returned.

Here is an example:

// The following returns code 200, yet the returned page is different from the acual page that is accessed via the browser

NOTE: To verify, I looked at the returned page vie PHP, and the source code of the page via the browser, and they are different.

I am not sure what the problem is. I am suspecting it might be someting to do with the argumets of wp_remote_get(), however, documentation about arguments is not clear.

Is it the arguments? If so, what and how to set those up? If not, what do you suspect the problem?


After reading the comment made by zendka in which he states "I get exactly the same content. I just checked again and compared the response body to the browser's page source. They're identical :)", I realized something very important.

The original URL that was used in my code in wp_remote_get() is http://www.aljazeera.net/news/international/2016/8/24/قتلى-ودمار-بزلزال-يضرب-وسط-إيطاليا

And the call would be as

// Causes problems

The URL that is used and originally described in this question was

// Runs successfully


  1. For the url to work with wp_remote_get() without causing any problem or errors, it should NOT be of the following format http://www.aljazeera.net/news/international/2016/8/24/قتلى-ودمار-بزلزال-يضرب-وسط-إيطاليا but rather it should be encoded as follows http://www.aljazeera.net/news/international/2016/8/24/%D9%82%D8%AA%D9%84%D9%89-%D9%88%D8%AF%D9%85%D8%A7%D8%B1-%D8%A8%D8%B2%D9%84%D8%B2%D8%A7%D9%84-%D9%8A%D8%B6%D8%B1%D8%A8-%D9%88%D8%B3%D8%B7-%D8%A5%D9%8A%D8%B7%D8%A7%D9%84%D9%8A%D8%A7.

  2. The reason wp_remote_get() is unable to work with http://www.aljazeera.net/news/international/2016/8/24/قتلى-ودمار-بزلزال-يضرب-وسط-إيطاليا is because it is unable to deal with this utf8 format. For it to work properly, only the path of the url should be converted to a percentage-encoded format. Moreover, any slash (/) character in the path should not be encoded, otherwise wp_remote_get() will not be able to find the page, and hence we get a success code (200) with 404 page content. Therefore, this format http://www.aljazeera.net/news/international/2016/8/24/%D9%82%D8%AA%D9%84%D9%89-%D9%88%D8%AF%D9%85%D8%A7%D8%B1-%D8%A8%D8%B2%D9%84%D8%B2%D8%A7%D9%84-%D9%8A%D8%B6%D8%B1%D8%A8-%D9%88%D8%B3%D8%B7-%D8%A5%D9%8A%D8%B7%D8%A7%D9%84%D9%8A%D8%A7 works correctly when passed to wp_remote_get().

  3. The reason I reported the formats incorrectly, is in the midst of my testing, I copied the url from Firefox and pasted it here. Firefox converts formatting automatically to the "correct" format. Details here: https://superuser.com/questions/480692/copying-unicode-symbols-from-firefox-address-bar-as-is

The solution

  1. Verify the URL has the correct percentage format

  2. Use the newly formatted url with wp_remote_get()

Helpful Resources






  • Not sure if this is a WordPress specific problem. Did you run a diff on those two versions?
    – birgire
    Commented Aug 24, 2016 at 8:59
  • Why not a WordPress question? I am using wp_remote_get()
    – Greeso
    Commented Aug 24, 2016 at 9:03
  • What I meant was that it's possible that part of the content could be dependent of the IP address, where your WordPress server could be in another country and might be in another target group?
    – birgire
    Commented Aug 24, 2016 at 9:09
  • I am sure this is not the case. Both my servers and browsers are in the same country.
    – Greeso
    Commented Aug 24, 2016 at 9:13
  • Oh, URL encoding! @Gresso, feel free to rephrase the question and post your finding as an answer. The right answer.
    – zendka
    Commented Aug 27, 2016 at 20:38

1 Answer 1


The HTTP request sent by wp_remote_get() is different from the one that a browser sends. E.g the user-agent is different (see documentation). Some websites respond differently based on this. The second argument of wp_remote_get() allows one to alter the request.

Websites might also respond differently depending on the IP or the number of requests received (usually by limiting or banning certain patterns).

By the way, I tested the page you mentioned and it works for me:

  • 1
    Are you sure you tested this page successfully? When I test the page, the return code is "success" (code 200), but the page that is returned is NOT the same as the actual page online. In other words, what I receive is a page different than the page I see in the browser. This is why I asked the question in the first place.
    – Greeso
    Commented Aug 24, 2016 at 19:06
  • I get exactly the same content. I just checked again and compared the response body to the browser's page source. They're identical :)
    – zendka
    Commented Aug 25, 2016 at 18:02
  • I was about to write to you. What you told me gave me a hint of what is going on. I will keep you posted.
    – Greeso
    Commented Aug 25, 2016 at 18:04
  • are you running the code on your localhost? (trying to figure out if it's the same IP as the browser)
    – zendka
    Commented Aug 25, 2016 at 18:04
  • I accepted your solution and gave you the reward because interacting with you helped me with the solution. Thanks.
    – Greeso
    Commented Aug 27, 2016 at 0:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.