I'd like to hear what others who are delivering complex non-blog solutions to clients with WordPress as a platform what they are using for automated Regression Testing?
For those not familiar with the term "regression testing" Wikipedia defines it as:
Regression testing is any type of software testing that seeks to uncover software errors after changes to the program (e.g. bugfixes or new functionality) have been made, by retesting the program. The intent of regression testing is to assure that a change, such as a bugfix, did not introduce new bugs.
More telling Wikipedia says the following which is exactly what I'm experiencing on a project right now:
Experience has shown that as software is fixed, emergence of new and/or reemergence of old faults is quite common. Sometimes reemergence occurs because a fix gets lost through poor revision control practices (or simple human error in revision control). Often, a fix for a problem will be "fragile" in that it fixes the problem in the narrow case where it was first observed but not in more general cases which may arise over the lifetime of the software. Frequently, a fix for a problem in one area inadvertently causes a software bug in another area. Finally, it is often the case that when some feature is redesigned, some of the same mistakes that were made in the original implementation of the feature were made in the redesign.
With the global nature of actions and filters I'm finding that complexity starts ramping as I add more client-requested functionality and it becomes hard to get a complex plugin stable, especially if it uses a lot of calls to
WP_Query and updates the database a lot.
The solution in my mind would be to set up regression testing with a series of "test cases" to comprise a "test suite." In concept it's not that hard when you are testing the HTML output of HTTP GET requests. But it gets a little more complicated when you have to test things when logged in via the admin console and/or to test jQuery interactions.
I'm setting this up as a community wiki in hopes we can gather best practices here but I'm really anxious to hear processes if any other WordPress professionals are using.