wp_remote_get() not retrieving pages properly

Question

I am using wp_remote_get() to retrieve different pages from the web. It is a news site.

Most of the time, this works successfully. However, sometimes, the request returns success (code 200), but the content of the page is differet from the page that is retrieved by a brwoser (the content of the page returned with this call is something along the lines of "page not found")

However if I use any browser to retrieve the page in question, the page is successfully returned.

Here is an example:

// The following returns code 200, yet the returned page is different from the acual page that is accessed via the browser
wp_remote_get("http://www.aljazeera.net/news/international/2016/8/24/%D9%82%D8%AA%D9%84%D9%89-%D9%88%D8%AF%D9%85%D8%A7%D8%B1-%D8%A8%D8%B2%D9%84%D8%B2%D8%A7%D9%84-%D9%8A%D8%B6%D8%B1%D8%A8-%D9%88%D8%B3%D8%B7-%D8%A5%D9%8A%D8%B7%D8%A7%D9%84%D9%8A%D8%A7");

NOTE: To verify, I looked at the returned page vie PHP, and the source code of the page via the browser, and they are different.

I am not sure what the problem is. I am suspecting it might be someting to do with the argumets of wp_remote_get(), however, documentation about arguments is not clear.

Is it the arguments? If so, what and how to set those up? If not, what do you suspect the problem?

---

IMPORTANT UPDATE AND SOLUTION

After reading the comment made by zendka in which he states "I get exactly the same content. I just checked again and compared the response body to the browser's page source. They're identical :)", I realized something very important.

The original URL that was used in my code in wp_remote_get() is http://www.aljazeera.net/news/international/2016/8/24/قتلى-ودمار-بزلزال-يضرب-وسط-إيطاليا

And the call would be as

// Causes problems
wp_remote_get("http://www.aljazeera.net/news/international/2016/8/24/قتلى-ودمار-بزلزال-يضرب-وسط-إيطاليا");

The URL that is used and originally described in this question was

// Runs successfully
wp_remote_get("http://www.aljazeera.net/news/international/2016/8/24/%D9%82%D8%AA%D9%84%D9%89-%D9%88%D8%AF%D9%85%D8%A7%D8%B1-%D8%A8%D8%B2%D9%84%D8%B2%D8%A7%D9%84-%D9%8A%D8%B6%D8%B1%D8%A8-%D9%88%D8%B3%D8%B7-%D8%A5%D9%8A%D8%B7%D8%A7%D9%84%D9%8A%D8%A7");

NOTE THE FOLLOWING:

1. For the url to work with wp_remote_get() without causing any problem or errors, it should NOT be of the following format http://www.aljazeera.net/news/international/2016/8/24/قتلى-ودمار-بزلزال-يضرب-وسط-إيطاليا but rather it should be encoded as follows http://www.aljazeera.net/news/international/2016/8/24/%D9%82%D8%AA%D9%84%D9%89-%D9%88%D8%AF%D9%85%D8%A7%D8%B1-%D8%A8%D8%B2%D9%84%D8%B2%D8%A7%D9%84-%D9%8A%D8%B6%D8%B1%D8%A8-%D9%88%D8%B3%D8%B7-%D8%A5%D9%8A%D8%B7%D8%A7%D9%84%D9%8A%D8%A7.

2. The reason wp_remote_get() is unable to work with http://www.aljazeera.net/news/international/2016/8/24/قتلى-ودمار-بزلزال-يضرب-وسط-إيطاليا is because it is unable to deal with this utf8 format. For it to work properly, only the path of the url should be converted to a percentage-encoded format. Moreover, any slash (/) character in the path should not be encoded, otherwise wp_remote_get() will not be able to find the page, and hence we get a success code (200) with 404 page content. Therefore, this format http://www.aljazeera.net/news/international/2016/8/24/%D9%82%D8%AA%D9%84%D9%89-%D9%88%D8%AF%D9%85%D8%A7%D8%B1-%D8%A8%D8%B2%D9%84%D8%B2%D8%A7%D9%84-%D9%8A%D8%B6%D8%B1%D8%A8-%D9%88%D8%B3%D8%B7-%D8%A5%D9%8A%D8%B7%D8%A7%D9%84%D9%8A%D8%A7 works correctly when passed to wp_remote_get().

3. The reason I reported the formats incorrectly, is in the midst of my testing, I copied the url from Firefox and pasted it here. Firefox converts formatting automatically to the "correct" format. Details here: https://superuser.com/questions/480692/copying-unicode-symbols-from-firefox-address-bar-as-is

The solution

1. Verify the URL has the correct percentage format

2. Use the newly formatted url with wp_remote_get()

Helpful Resources

https://github.com/neitanod/forceutf8

https://stackoverflow.com/questions/910793/detect-encoding-and-make-everything-utf-8

https://stackoverflow.com/questions/2742852/unicode-characters-in-urls

https://stackoverflow.com/questions/9831077/how-to-url-encode-only-non-ascii-symbols-of-url-in-php-but-leave-reserved-symbo

https://stackoverflow.com/questions/25465114/php-str-replace-not-working-with-special-chars

Answer 1

The HTTP request sent by wp_remote_get() is different from the one that a browser sends. E.g the user-agent is different (see documentation). Some websites respond differently based on this. The second argument of wp_remote_get() allows one to alter the request.

Websites might also respond differently depending on the IP or the number of requests received (usually by limiting or banning certain patterns).

By the way, I tested the page you mentioned and it works for me:

wp_remote_get("http://www.aljazeera.net/news/international/2016/8/24/%D9%82%D8%AA%D9%84%D9%89-%D9%88%D8%AF%D9%85%D8%A7%D8%B1-%D8%A8%D8%B2%D9%84%D8%B2%D8%A7%D9%84-%D9%8A%D8%B6%D8%B1%D8%A8-%D9%88%D8%B3%D8%B7-%D8%A5%D9%8A%D8%B7%D8%A7%D9%84%D9%8A%D8%A7");