Take the 2-minute tour ×
WordPress Development Stack Exchange is a question and answer site for WordPress developers and administrators. It's 100% free, no registration required.

I would like to retrieve all posts of a blog via the XML-RPC API of Wordpress.

There are the methods of blogger.getRecentPosts and metaWeblog.getRecentPosts which - given a sufficiently high value as number of posts (or -1) in theory should return all posts.

However, this does not work for very large blogs or very weak servers that cannot hold the whole blog in memory. In that case, these functions will not return anything at best or throw an error into the response XML.

A solution would be to retrieve smaller chunks of, e.g. 50, posts at a time and put it all together on the receiving side. For this to work one would need to specify an offset for the posts to get. I was not able to find a way to specify such an offset in the documented API.

Is there any way to make this work, either by specifying an offset or by using other than the methods mentioned above?

I'm not looking for a description on how to write a plugin or modify Wordpress itself in any way. I can do that, sure, but I'm talking about authorized retrieval of data of arbitrary Wordpress blogs.

Edit: I've opened a trac ticket at Wordpress with a suggestion for solution: http://core.trac.wordpress.org/ticket/16316

share|improve this question
    
Before we answer, could you please explain why you want to get all the posts? This is very similar to content scraping ... which you're not likely to get any help with. –  EAMann Jan 20 '11 at 1:07
    
@EAMann since question puts it as authorized retrieval of data I don't think this is about nefarious scraping. Does XML-RPC work for public access at all? I am not sure. –  Rarst Jan 20 '11 at 6:57
    
XML-RPC requires authorized access with username and password. As I mentioned, this is about authorized access. The reason is that we're working on a blogbook software at feedfabrik.com/bookfabrik –  kongo09 Jan 20 '11 at 11:42
add comment

5 Answers

up vote 0 down vote accepted

According to topic in official forums [xmlrpc] How to get posts with offset?

The existing XML-RPC APIs don't really provide a way for collecting all of the post data right now. (Joseph Scott)

Topic is somewhat old and I am not aware if there were some changes since, but from quick look at source it doesn't seem so.

share|improve this answer
    
Thank you, this link to the forum is helpful - in that there is probably no help - apart from doing what people don't like: write a screenscraper. –  kongo09 Jan 20 '11 at 11:45
add comment

Let me apologize for the initial question I had about your motives. I see a lot of "how can I remotely retrieve all posts from another blog" questions and immediately assume there is nefarious intent because, 9 times out of 10, there is. That said, your purposes seem very straight-forward and respectable.

Currently, there is no way to "chunk" the XML return of any of the three requests you've mentioned. When I got up this morning, though, I saw you've proposed this as a feature enhancement through Trac. This definitely won't make it in to Wordpress 3.1, so you'll likely be waiting a few months (or longer) before any submitted patches make it into core. But this is a good start.

In the mean time, remember that the XML-RPC API is extensible. While there is no way to receive "chunks" in the existing API, you can always add your own method. This is actually the best way to get a patch in core - create your own method, make sure it works, and submit the patch back to Trac.

My guess is that your method would be very similar to metaWeblog.getRecentPosts, but would be named a bit better ... perhaps wp.getPagedPosts. You could accept all the same parameters, but add one: "pagenumber". This way you could set the request to return 50 posts at a time and progressively walk through the collection.

To add your method, you hook into the xmlrpc_methods filter:

function xml_add_method( $methods ) {
    $methods['wp.getPagedPosts'] = 'wp_getPagedPosts';
    return $methods;
}
add_filter( 'xmlrpc_methods', 'xml_add_method');

Then add your callback function:

function wp_getPagedPosts($args) {

    // $this->escape($args);    //<-- This is called by native XML-RPC methods to sanitize passed arrays for the database.

    $blog_ID     = (int) $args[0];
    $username  = $args[1];
    $password   = $args[2];
    if ( isset( $args[3] ) )
        $query = array( 'numberposts' => absint( $args[3] ) );
    else
        $query = array();

    if ( !$user = $this->login($username, $password) )
        return $this->error;

    do_action('xmlrpc_call', 'wp.getPagedPosts');

    //... get a list of posts and generate your XML-RPC return ...

}

Remember, this is code you'd place in an external plug-in file or a functions.php file with your theme to support the additional XML-RPC request. There aren't any existing methods to handle this, so you're stuck with writing your own. But if you do it once, and do it well, and submit it back to Trac ... it could become Core, then you wouldn't have to do it again.

share|improve this answer
    
I appreciate your detailed answer but I was not looking for code for a plugin. In fact, I have extended the XML-RPC API of Wordpress already for our Feedfabrik Blog Printing pluging at wordpress.org/extend/plugins/feedfabrik The problem is that many people would like to use this service without installing a plugin first - so I feel patching the core is the only way to go. But thanks for your nice example. –  kongo09 Jan 21 '11 at 10:52
add comment

A simple way of grabbing all posts is to first try running the getRecentPosts method and retrieve only 1 post (the most recent is what will get returned) and using this post's ID to simply loop back and grab each post consecutively with the getPost method. if you're having issues with the amount of data being retrieved in one call then this should solve your problem. You could even alter this to grab chunks by changing your $count to 5 or 10 or what-have-you.

Here's a severely stripped down example (assuming you are familiar with the two methods in PHP, hopefully you get the idea..):

<?php
include('your_xmlrpc_functions_for_getRecentPosts_and_getPosts.php');
$count = 1;
$dataArray = getRecentPosts($appkey, $blogid, $user, $pass, $count);
$startID = $dataArray[0][postid];
for($i=$startID;$i>0;$i--) {
 $postInfoArray[] = getPost($appkey, $blogid, $user, $pass, $i);
 //add in whatever other functionality for each post here (maybe a time delay or something)
}
?>

Hope this helps someone :)

share|improve this answer
1  
Unfortunately, this only works for the smallest of blogs (which probably also work with a full retrieval in one go). For large blogs, it would take an extremely long time to get the content by single posts. In addition, once you've made the first call to getRecentPosts with a count, you cannot use count together with another starting point to grab chunks as you describe. This would be perfect, but the function doesn't exist. –  kongo09 Sep 20 '11 at 20:49
    
kongo90 -- my solution gets the job done for a typical blog size. I ran my script on a blog of 2500 posts and it took around 8 minutes to process which is perfectly acceptable in a normal situation where immediate gratification is not a primary necessity. Even if it took an hour to process, using my script once a day is acceptable. But if you're working with something bigger than, say, 500000 posts then you shouldn't be using xmlrpc in the first place. The xml-rpc protocol is meant for simple transactions that don't stress the mysql server too hard, that's why more complex methods have not bee –  Zekar Kovacs Sep 21 '11 at 1:14
add comment

It's easy. Just use metaWeblog.getRecentPosts or mt.getRecentPostTitles and set limit to PHP_INT_MAX. If you set it to 0 it returns a maximum of posts you set to display on homepage (10 usually). My Wp Remote Control Library does this with great ease. See the Basic Snippets.

// Getting all posts as full or light items
$all_posts = $wpapi->getRecentPosts(PHP_INT_MAX);
$all_post_titles = $wpapi->getRecentPostsList(PHP_INT_MAX);

Regards.

share|improve this answer
    
Just to further explain. You can't get posts in chunks, unless you go directly in the database or use my rSQL plugin that facilitates MySQL queries and access remotely through XMLRPC. –  EarnestoDev Oct 10 '11 at 13:34
    
Thanks for your reply. Unfortunately, as you added in your comment, it is not easy at all but rather impossible. getRecentPosts only works for smaller blogs. Big blogs will blow the memory limit of PHP. That's why I need to chunk it in the first place. Also, I explicitely asked for a solution without plugins as I have a plugin myself already. –  kongo09 Oct 10 '11 at 13:59
    
mt.getRecentPostTitles returns lighter variants of the data. Try it! –  EarnestoDev Oct 10 '11 at 14:07
    
getRecentPostTitles returns the titles (and a bit) so that is in a no way an alternative. –  kongo09 Oct 10 '11 at 15:53
add comment

wp.getPosts now offers the ability to request an offset.

http://codex.wordpress.org/XML-RPC_WordPress_API/Posts#wp.getPost

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.