Bulk insert posts is really slow and skips entries

Question

I'm writing a plugin to parse a bunch of XML files with the NewsML standard and insert them into the database as custom post type. Everything works fine and as intended, but especially at the first activation of the plugin, there are many files to import and this takes a long time. I'm talking about up to 2000 files.

The parsing process takes about 20-30 seconds which is ok for me but the whole import process takes up to 6 minutes or more and I don't think that it is the best practice to raise the execution time limit.

Sometimes my script just skips all files after the execution time limit and shows a blank page, in my last try I got an php error and sometimes it just works. (I assume it works when it does not reach the time limit).

Here is my code to import the array of entries in the database. Since I began developing wordpress plugins just 6 weeks ago I know that it's not the best code you've ever seen. https://gist.github.com/inrebinfo/0dba270e7fa1bca60a79

So I'm looking for a solution with 2 goals:
1: Don't skip any files. I don't know how to bypass the execution time limit. (Maybe split the array and import the single parts?)
2: Speed the process up. The 20 seconds for the parsing is okay, but the insert takes way to long in my opinion.

I would be happy for any advice and tips.

Answer 1

A queue may be your solution.

The individual items that should be imported can be put into a queue (e.g. in a database table) which can then be worked on 1 by 1 during the WP Cron process.

See how other plug-ins handle job queues

Hope that helps.