Issues with downloading a file with wp_remote_get()

Question

I'm using wp_remote_get() to download a remote file with the following URL:

http://feeds.paidonresults.net/1.0?affiliateid=34069&feedid=M742&template=CSV&CSVSeperator=comma&field_names=ProductID,ProductName,ProductPrice,SummaryDescription,ProductDescription,AffiliateURL,ImageURL100by100,ImageURL200by200,ProductAddedDate,Category,MerchantName

Using http://web-sniffer.net/ the headers are as follows:

Connect to 80.249.99.4 on port 80 ... ok

GET /1.0?affiliateid=34069&feedid=M742&template=CSV&CSVSeperator=comma&field_names=ProductID%2CProductName%2CProductPrice%2CSummaryDescription%2CProductDescription%2CAffiliateURL%2CImageURL100by100%2CImageURL200by200%2CProductAddedDate%2CCategory%2CMerchantName HTTP/1.1[CRLF]
Host: feeds.paidonresults.net[CRLF]
Connection: close[CRLF]
User-Agent: Web-sniffer/1.0.37 (+http://web-sniffer.net/)[CRLF]
Accept-Encoding: gzip[CRLF]
Accept-Charset: ISO-8859-1,UTF-8;q=0.7,*;q=0.7[CRLF]
Cache-Control: no-cache[CRLF]
Accept-Language: de,en;q=0.7,en-us;q=0.3[CRLF]
Referer: http://web-sniffer.net/[CRLF]
[CRLF]

Which is great.

However, using wp_remote_get() I am receiving the following:

Array
(
[headers] => Array
    (
        [date] => Mon, 07 May 2012 19:12:56 GMT
        [server] => Apache/2.2.22 (Unix)
        [p3p] => policyref="http://www.paidonresults.com/w3c/p3p.xml",CP="CAO DSP COR LAW DEVa TAIa OUR BUS UNI ADMa CURa PHY ONL PUR COM NAV DEM STA"
        [connection] => close
        [content-type] => text/html
    )

[body] => Fatal Error 
[response] => Array
    (
        [code] => 200
        [message] => OK
    )

[cookies] => Array
    (
    )

[filename] => 
)

I know there is an issue with a wp_remote_get() and a remote redirect, but I've used file_get_contents() and the purpose built function here but I'm still getting the same body fatal error response

Does anybody know a way around this?

Answer 1

I am retrieving this URL ok on my test stack, which means that either:

1. Remote server was having temporary issues.
2. Your server has configuration or other issues.

I would try to install Core Control plugin, force different HTTP transports and see if that helps.

Answer 2

Finally solved the problem

I was using wp_kses() to sanitize the data and it was mangling up the URL

It was the totally wrong Wordpress function to use for this purpose

What confused me is the response from the remote server. It the URL wasn't correct that I was expecting a non-200 header response.

Well that is 4 hours of my life I won't get back :(