Take the 2-minute tour ×
WordPress Development Stack Exchange is a question and answer site for WordPress developers and administrators. It's 100% free, no registration required.

Hi all,

I'd like to hear what others who are delivering complex non-blog solutions to clients with WordPress as a platform what they are using for automated Regression Testing?

For those not familiar with the term "regression testing" Wikipedia defines it as:

Regression testing is any type of software testing that seeks to uncover software errors after changes to the program (e.g. bugfixes or new functionality) have been made, by retesting the program. The intent of regression testing is to assure that a change, such as a bugfix, did not introduce new bugs.

More telling Wikipedia says the following which is exactly what I'm experiencing on a project right now:

Experience has shown that as software is fixed, emergence of new and/or reemergence of old faults is quite common. Sometimes reemergence occurs because a fix gets lost through poor revision control practices (or simple human error in revision control). Often, a fix for a problem will be "fragile" in that it fixes the problem in the narrow case where it was first observed but not in more general cases which may arise over the lifetime of the software. Frequently, a fix for a problem in one area inadvertently causes a software bug in another area. Finally, it is often the case that when some feature is redesigned, some of the same mistakes that were made in the original implementation of the feature were made in the redesign.

With the global nature of actions and filters I'm finding that complexity starts ramping as I add more client-requested functionality and it becomes hard to get a complex plugin stable, especially if it uses a lot of calls to WP_Query and updates the database a lot.

The solution in my mind would be to set up regression testing with a series of "test cases" to comprise a "test suite." In concept it's not that hard when you are testing the HTML output of HTTP GET requests. But it gets a little more complicated when you have to test things when logged in via the admin console and/or to test jQuery interactions.

I'm setting this up as a community wiki in hopes we can gather best practices here but I'm really anxious to hear processes if any other WordPress professionals are using.

share|improve this question
    
I assume you are talking about testing your own code (themes/plugins)? When you create new code, or when you update "the environment" (WP, other plugins)? Or both? I think Pro Webmasters could also contain good advice on how to test web apps (Selenium and stuff) - maybe cross-posting is a good idea? –  Jan Fabry Dec 2 '10 at 9:15
    
@Jan Fabry - Yes, testing my own code. Good idea about cross-posting, I'll do that soon. –  MikeSchinkel Dec 2 '10 at 10:28

2 Answers 2

up vote 8 down vote accepted

PHPUnit would come to mind, if the WP test suite wasn't so broken, and if WP had been designed and written in a way that it could actually be tested properly. ;-)

More seriously, you can test your plugins all you want from their functional standpoint with unit tests and the like. The issue is that these tests won't guarantee that they'll catch subtle chances introduced by WP upgrades, let alone that they'll continue to work once plugged into a customized WP install.

Among the colorful things I've seen happen:

  • A subtle change in the WP API affects your plugin's functionality, e.g. the hook you're on used to get a term id and it's now getting a term taxonomy id. (Chances are good that your test terms conveniently have the same id for both).

  • A subtle change in the WP API results in your receiving a WP_Error object instead of the previously expected value of false as bad input.

  • Your plugin is added from within the mu-plugins folder, resulting in a subtly different code flow.

  • Your plugin worked fine until memcached or some other persistent store got enabled.

  • Your plugin worked fine until the despised switch_to_blog() got called.

  • A plugin changes the hook which it resides on when called, and unknowingly interrupts it as a side effect.

  • A plugin (un?)knowingly messes around with your input or output data to the point where things look broken even though you're not at fault.

I could expand the list on and on, but those would be the key items that broke my own plugins. The two items are arguably catchable with unit tests. The next two as well, if you're patient enough, but I'd argue that WP should not change the way things work when they occur. No amount of testing will work around the buggy implementation of switch_to_blog(). And the last two are hopelessly untestable.

Oh, and... don't even get me started on the onslaught of attachments, auto drafts, revisions, menu items and what not that end up stored in the posts table.

Good luck... :-)

share|improve this answer
1  
Nice answer, thanks for the all details you cover. FWIW I'm looking more for "regression" testing than "unit" testing. I know there's lots of overlap but my biggest issue right now is verifying that the website doesn't break. Yes unit testing a plugin can catch most issues but it takes a lot more time and effort to get full coverage on unit tests (which means I'll probably not get full coverage) whereas the full page testing is much less fine grained in requirements. –  MikeSchinkel Dec 2 '10 at 12:20
1  
Actually, note that there are tools in certain frameworks (Symfony2 and Li3 to name but two) that allow to test an actual site, using a fictive browser. The components in question are reusable for other things. Thus, you could actually manipulate your site's admin screens and verify that what you're doing has the expected results. –  Denis Dec 2 '10 at 12:49
    
Thanks again! –  MikeSchinkel Dec 4 '10 at 4:31

You should strongly consider Selenium.

It allows you to record actions (e.g. entering data into a form, clicking a link) and then you can perform assertions. It also integrates with PHPUnit. I highly recommend checking out the two minute demo.

share|improve this answer
    
Thanks for the suggestion, I'd hear of it before. Have you actually used for WordPress projects? Just curious. –  MikeSchinkel Dec 4 '10 at 4:31
    
Yes. I have used it for testing a plugin I am working on. In a previous life, we used it to test EDC apps for clinical research. –  Ethan Seifert Dec 4 '10 at 18:47

protected by toscho Aug 3 '12 at 7:51

Thank you for your interest in this question. Because it has attracted low-quality answers, posting an answer now requires 10 reputation on this site.

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.