23

Hi all,

I'd like to hear what others who are delivering complex non-blog solutions to clients with WordPress as a platform what they are using for automated Regression Testing?

For those not familiar with the term "regression testing" Wikipedia defines it as:

Regression testing is any type of software testing that seeks to uncover software errors after changes to the program (e.g. bugfixes or new functionality) have been made, by retesting the program. The intent of regression testing is to assure that a change, such as a bugfix, did not introduce new bugs.

More telling Wikipedia says the following which is exactly what I'm experiencing on a project right now:

Experience has shown that as software is fixed, emergence of new and/or reemergence of old faults is quite common. Sometimes reemergence occurs because a fix gets lost through poor revision control practices (or simple human error in revision control). Often, a fix for a problem will be "fragile" in that it fixes the problem in the narrow case where it was first observed but not in more general cases which may arise over the lifetime of the software. Frequently, a fix for a problem in one area inadvertently causes a software bug in another area. Finally, it is often the case that when some feature is redesigned, some of the same mistakes that were made in the original implementation of the feature were made in the redesign.

With the global nature of actions and filters I'm finding that complexity starts ramping as I add more client-requested functionality and it becomes hard to get a complex plugin stable, especially if it uses a lot of calls to WP_Query and updates the database a lot.

The solution in my mind would be to set up regression testing with a series of "test cases" to comprise a "test suite." In concept it's not that hard when you are testing the HTML output of HTTP GET requests. But it gets a little more complicated when you have to test things when logged in via the admin console and/or to test jQuery interactions.

I'm setting this up as a community wiki in hopes we can gather best practices here but I'm really anxious to hear processes if any other WordPress professionals are using.

2
  • I assume you are talking about testing your own code (themes/plugins)? When you create new code, or when you update "the environment" (WP, other plugins)? Or both? I think Pro Webmasters could also contain good advice on how to test web apps (Selenium and stuff) - maybe cross-posting is a good idea?
    – Jan Fabry
    Dec 2, 2010 at 9:15
  • @Jan Fabry - Yes, testing my own code. Good idea about cross-posting, I'll do that soon. Dec 2, 2010 at 10:28

3 Answers 3

10

PHPUnit would come to mind, if the WP test suite wasn't so broken, and if WP had been designed and written in a way that it could actually be tested properly. ;-)

More seriously, you can test your plugins all you want from their functional standpoint with unit tests and the like. The issue is that these tests won't guarantee that they'll catch subtle chances introduced by WP upgrades, let alone that they'll continue to work once plugged into a customized WP install.

Among the colorful things I've seen happen:

  • A subtle change in the WP API affects your plugin's functionality, e.g. the hook you're on used to get a term id and it's now getting a term taxonomy id. (Chances are good that your test terms conveniently have the same id for both).

  • A subtle change in the WP API results in your receiving a WP_Error object instead of the previously expected value of false as bad input.

  • Your plugin is added from within the mu-plugins folder, resulting in a subtly different code flow.

  • Your plugin worked fine until memcached or some other persistent store got enabled.

  • Your plugin worked fine until the despised switch_to_blog() got called.

  • A plugin changes the hook which it resides on when called, and unknowingly interrupts it as a side effect.

  • A plugin (un?)knowingly messes around with your input or output data to the point where things look broken even though you're not at fault.

I could expand the list on and on, but those would be the key items that broke my own plugins. The two items are arguably catchable with unit tests. The next two as well, if you're patient enough, but I'd argue that WP should not change the way things work when they occur. No amount of testing will work around the buggy implementation of switch_to_blog(). And the last two are hopelessly untestable.

Oh, and... don't even get me started on the onslaught of attachments, auto drafts, revisions, menu items and what not that end up stored in the posts table.

Good luck... :-)

2
  • 2
    Nice answer, thanks for the all details you cover. FWIW I'm looking more for "regression" testing than "unit" testing. I know there's lots of overlap but my biggest issue right now is verifying that the website doesn't break. Yes unit testing a plugin can catch most issues but it takes a lot more time and effort to get full coverage on unit tests (which means I'll probably not get full coverage) whereas the full page testing is much less fine grained in requirements. Dec 2, 2010 at 12:20
  • 1
    Actually, note that there are tools in certain frameworks (Symfony2 and Li3 to name but two) that allow to test an actual site, using a fictive browser. The components in question are reusable for other things. Thus, you could actually manipulate your site's admin screens and verify that what you're doing has the expected results. Dec 2, 2010 at 12:49
7

You should strongly consider Selenium.

It allows you to record actions (e.g. entering data into a form, clicking a link) and then you can perform assertions. It also integrates with PHPUnit. I highly recommend checking out the two minute demo.

2
  • Thanks for the suggestion, I'd hear of it before. Have you actually used for WordPress projects? Just curious. Dec 4, 2010 at 4:31
  • Yes. I have used it for testing a plugin I am working on. In a previous life, we used it to test EDC apps for clinical research. Dec 4, 2010 at 18:47
1

Selenium is probably usable but I think that in modern times you will find Codeception to be better and easier to use. For the simplest visual regression testing, it even has an extension that will take screenshots and automatically compare them for you.

Of course, Codeception WebDriver tests can go further and perform functional regression tests. You can fill out and submit forms, click buttons and links on the site, execute any JS, etc. You can use a real-life browser like Firefox or Chrome in your tests, or you can perform headless testing with PhantomJS. That means that you can even run the WebDriver tests for your plugin as part of its build process on Travis CI if you want to.

There are even several WordPress-specific libraries to help you get started:

1
  • 1
    Selenium and Codeception aren't exclusive. You can use WP-Browser to drive Selenium [which drives an actual browser like Chrome], Phantom [which is a non-gui browser with JS support], or even the PHPBrowser which is a dumb curl browser [very fast but no JS. i.e. API tests]. WP-Browser can drive any of them. Aug 8, 2017 at 4:32

Not the answer you're looking for? Browse other questions tagged or ask your own question.