Over the last few months, I spent a lot of cold winter evening hours looking into porting ancient PHPUnit 4.x test suites over to PHPUnit 9.x. The test suites, you guessed it, belong to the Horde framework. Horde actually does not just use phpunit but wraps it into its own testing library horde/test. The full details are explained on the wiki. Horde provides multiple ways to actually run the unit tests, either through its components helper or through AllTests.php or through calling individual tests with phpunit. While horde/test adds value and allows simplifying some test scenarios, it also has its own problems.
I’m not saying it didn’t make sense back when this was built. The original alpha release of horde/test dates back to 2011. PHP 5.4 was not yet released, composer would see its initial release in 2012, PEAR was slowly getting old and PEAR2 was not yet officially a dead cow. It was a totally different ecosystem back then. However, times have changed.
As of 2021, phpunit has evolved into a relatively fast moving target and while its primary author Sebastian Bergmann does not break backward compatibility for the sake of it, he is not shy of doing it either. Some parts of phpunit which are relevant for integration may become unavailable in the next major release. Some of PHPUnit’s core classes are clearly marked as internal.
The Horde Test library adds some mandatory boilerplate to phpunit: Each library and app has its own mandatory bootstrap.php file calling into the test library’s bootstrap class and may have an additional autoload file. Also, a test suite should come with an AllTests.php file calling into the test package’s AllTests class and in turn using Horde_Test_AllTests_TestRunner – which extends PHPUnit\Runner\BaseTestRunner but wait…
https://github.com/sebastianbergmann/phpunit/blob/9.5/src/Runner/BaseTestRunner.php
/**
* @internal This class is not covered by the backward compatibility promise for PHPUnit
*/
abstract class BaseTestRunnerTrouble ahead, maybe. I am not joking. The master branch which will become PHPUnit 10 does not even have that file. We can find a solution for that, sure. But maybe we should not. As of 2021, the horde/test suite contains valuable helpers and extensions to PHPUnit, but none of these really need to hook into phpunit’s core that deep anymore.
Let’s look into different issues covered by that code.
Autoloading and dependency setup
Back in 2011 it made sense to have some glue which combines autoloading with Horde’s and PEAR’s notions of how things should be organized in a file system. That is no longer a core concern. Everybody and his dog use composer and composer’s autoloader and either do PSR-0 or PSR-4 autoloading schemes. Horde 6 will be delivered via composer, does – optionally, partially – break with some older ideas of organisation, brings some PSR-4 code and a lot of PSR-0 code. In short, there is very limited need for a custom autoloading scheme. The test suite should just rely on some autoloading being setup, by whoever or whatever. It should not address this concern beyond the means already provided by phpunit itself. Code should be accessed through checking if class names are either present or loadable, not by assuming some files are in some location. Of course, this is a little simplified – at its core, all autoloading depends on something being in some wellknown location. We should simply rely on the default solution until it is not possible for a specific case – and then address that. Even if we accept the notion of a vendor dir and an autoload file in vendor/autoload.php, we could provision it via horde/components or horde/git-tools or some horde/test utility for the rare cases where just using composer would not be appropriate.
Setting up complex test cases
Tests are supposed to be simple. They should focus on one unit under test and keep as much of the ecosystem as possible out of the picture. Dependencies should be stubbed or mocked. Databases and I/O, especially network traffic, should be substituted. A majority of test cases should be modeled with just the library’s own code and PHPUnit’s mocking facilities, maybe depending on some interfaces from some dependency.
But sometimes it’s not so easy. Sometimes the unit under test actually IS code interfacing with databases and their subtle differences. Sometimes the code needs to interact with a non-trivial amount of configuration and dependencies. Especially code which interacts with the framework’s core services or which couples the application’s subsystems may not be easy to mock. It makes sense to provide a simplified pretend environment for such test subjects. But that should be opt-in on a case by case basis. It should not be a mandatory tie-in of additional code and boilerplate for even the most trivial test cases.
Integration with tools
horde/components and some other tools wrap phpunit and other utilities. But as the ecosystem changes, the benefits are shrinking. Github Actions and Gitlab CI have become popular platforms with many ready-made CI tools available to use. It is no longer necessary to run your own Hudson or Jenkins and build your own automation frameworks. While it is still nice to have a short way to run a test suite including dead code and copy/paste detection, coding style fixers etc without the need to check anything into the SCM or even create a commit, there’s little incentive for maintaining a deep integration into your own runtime code. In the end, all testing and quality assurance automation aims to make you deliver safe, stable code as fast as possible. Spending ours writing automated tests makes sense and may safe you from hours of debugging and frustration. Spending hours writing or fixing a deep integration into some test tool which simply does not want to be integrated? Less so. It should be kept to a minimum. There’s a reason why the preferred delivery of some of these tools is not composer but phar.
What can you do?
So what would a new test toolchain look like? Unit tests and even integration tests should rely on a default, external autoloading solution. In the absense of configuration, this should be vendor/autoload.php, however it is delivered.
Unit tests should by default just run off PHPUnit and the library’s code. Maybe it makes sense to provide the most widespread interfaces without actually having to install their backing code. Why not, as long as it does not create manual effort. Any framework-specific recurring need should be addressed by opt-in code provided as traits or helper classes, available through default development-time autoloading sources. Fringe tests relying on specific infrastructure should skip without failing. Configurable tests should run out of the box in a useful default configuration if it makes sense. These changes can be created in an incremental, opt-in fashion with very limited BC breaks. This is good as nobody has time to waste on large scale transformation. Remember, it’s all about your code, the tests are just a useful tool.