The heart of SimpleTest is a testing framework built around test case classes. These are written as extensions of base test case classes, each extended with methods that actually contain test code. Top level test scripts then invoke the run() methods on every one of these test cases in order. Each test method is written to invoke various assertions that the developer expects to be true such as assertEqual(). If the expectation is correct, then a successful result is dispatched to the observing test reporter, but any failure triggers an alert and a description of the mismatch.
A test case looks like this...
class MyTestCase extends UnitTestCase { function MyTestCase() { $this->UnitTestCase(); } function testLog() { $log = new Log("my.log"); $log->message("Hello"); $this->assertTrue(file_exists("my.log")); } }
These tools are designed for the developer. Tests are written in the PHP language itself more or less as the application itself is built. The advantage of using PHP itself as the testing language is that there are no new languages to learn, testing can start straight away, and the developer can test any part of the code. Basically, all parts that can be accessed by the application code can also be accessed by the test code if they are in the same language.
The simplest type of test case is the UnitTestCase. This class of test case includes standard tests for equality, references and pattern matching. All these test the typical expectations of what you would expect the result of a function or method to be. This is by far the most common type of test in the daily routine of development, making up about 95% of test cases.
The top level task of a web application though is not to produce correct output from it's methods and objects, but to generate web pages. The WebTestCase class tests web pages. It simulates a web browser requesting a page, complete with cookies and GET/POST parameters. With this type of test case, the developer can assert that information is present in the page and that forms and sessions are handled correctly.
A web test case looks like this...
class MySiteTest extends WebTestCase { function MySiteTest() { $this->WebTestCase(); } function testHomePage() { $this->get("http://www.my-site.com/index.php"); $this->assertTitle("My Home Page"); $this->clickLink("Contact"); $this->assertTitle("Contact me"); $this->assertWantedPattern("/Email me at/"); } }
SimpleTest has a monthly release cycle. The following is a very rough outline of upcoming features and their expected point of release. I am afraid it is liable to change without warning as meeting the milestones rather depends on time available. Green stuff has been coded, but not necessarily released yet. If you have a pressing need for a green but unreleased feature then you should check-out the code from sourceforge CVS directly.
Feature | Description | Release |
---|---|---|
Unit test case | Core test case class and assertions | Done |
Html display | Simplest possible display | Done |
Autoloading of test cases | Reading a file with test cases and loading them into a group test automatically | Done |
Mock objects code generator | Objects capable of simulating other objects removing test dependencies | Done |
Server stubs | Mocks without expectations to be used outside of test cases, e.g. for prototyping | Done |
Integration of other unit testers | The ability to read and simulate test cases from PHPUnit and PEAR::PhpUnit | Done |
Web test case | Basic pattern matching of fetched pages | Done |
HTML parsing of pages | Allows link following and title tag matching | Done |
Partial mocks | Mocking parts of a class for testing less than a class or for complex simulations | Done |
Web cookie handling | Correct handling of cookies when fetching pages | Done |
Following redirects | Page fetching automatically follows 300 redirects | Done |
Form parsing | Ability to submit simple forms and read default form values | Done |
Command line interface | Test display without the need of a web browser | Done |
Exposure of expectation classes | Can create precise tests with mocks as well as test cases | Done |
XML output and parsing | Allows multi host testing and the integration of acceptance testing extensions | Done |
Command line test case | Allows testing of utilities and file handling | Done |
PHP Documentor compatibility | Fully generated class level documentation | Done |
Browser interface | Exposure of lower level web browser interface for more detailed test cases | Done |
HTTP authentication | Fetching protected web pages with basic authentication only | Done |
Browser navigation buttons | Back, forward and retry | Done |
SSL support | Can connect to https: pages | Done |
Proxy support | Can connect via. common proxies | Done |
Frames support | Handling of frames in web test cases | 1.0 |
Improved display | Better web GUI with tree display of test cases | 1.1 |
Localisation | Messages abstracted and code generated from XML | 1.1 |
The wish list includes integration with other test systems including JUnit and XsltUnit, an Eclipse plug-in, SOAP testing and HTTP 1.1 support. In reality I doubt these will ever happen.
Process is at least as important as tools. The type of process that makes the heaviest use of a developer's testing tool is of course Extreme Programming. This is one of the Agile Methodologies which combine various practices to "flatten the cost curve" of software development. More extreme still is Test Driven Development, where you very strictly adhere to the rule of no coding until you have a test. If your more of a planner or believe that experience trumps evolution, you may prefer the RUP approach. I haven't tried it, but even I can see that you will need test tools (see figure 9). Better yet, combine them into dX and then turn your head upside down.
Most unit testers clone JUnit to some degree, as far as the interface at least. There is a wealth of information on the JUnit site including the FAQ which contains plenty of general advice on testing. Once you get bitten by the bug you will certainly appreciate the phrase test infected coined by Eric Gamma. If you are still reviewing which unit tester to use the main choices are PHPUnit and Pear PHP::PHPUnit. They currently lack a lot of features found in SimpleTest, but the PEAR version at least is due for a major upgrade for PHP5.
Library writers don't seem to ship tests with their code very often which is a shame. Library code that includes tests can be more safely refactored and the test code can act as additional documentation in a fairly standard form. This can save trawling the source code for clues when problems occour, especially when upgrading such a library. If you are hunting for testing examples in PHP libraries then there are a few exceptions. The ISMO library for PHP uses the PEAR unit tester and also includes a clever trick for mocking PHP functions. Libraries using SimpleTest for their unit testing include WACT and PEAR::XML_HTMLSax.
There is currently a sad lack of material on mock objects, which is a shame as unit testing without them is a lot more work. The original mock objects paper is very Java focused, but still worth a read. As a new technology there are plenty of discussions and debate on how to use mocks, often on Wikis such as Extreme Tuesday or www.mockobjects.com or the original C2 Wiki. Injecting mocks into a class is the main area of debate for which this paper on IBM makes a good starting point.
There are plenty of web testing tools, but most are written in Java and tutorials and advice are rather thin on the ground. The only hope is to look at the documentation for HTTPUnit, HTMLUnit or JWebUnit and hope for clues. One interesting alternative is ITP. This framework uses XML to write the test cases, which is handy if your Java knowledge is limited (like mine). It still needs Java to run.
For coarse benchmarking you can use ab, the Apache Benchmark, or a number of other Jakarta testing tools. Apart from ab, you will probably have to have Java(tm) installed to use these alternatives. If you are not using Apache then you will have to read through your web server documentation or find a way to run your scripts from the command line to time them. End to end page timings are a very effective return on the time spent optimising. Once a page appears to be slow, breaking down the task timings to find bottlenecks can be a real challenge.
For fine profiling of the PHP code base itself there are two alternatives, although both have install issues. The first is apd (advanced PHP debugger) from the PEAR stable. The profiler comes as part of this debugging tool, but can be very slow to process the gathered data. The alternative XDebug comes highly recommended, but I have not tried this tool.
One very useful source of information though is you.
Yes, that's right, you!
Almost everything you discover in the PHP testing field will be pretty new even if perhaps it was also recently disovered by somebody else. If you have an opinion on fine grain testing of web applications, I would love to hear from you and so would everyone else. Together we can grope towards better practices and coach newcomers up to our self imposed coding standard - the one that says the code must at least work. I'll happily place any accumulated wisdom in this site manually until I can start a Blog, and I will of course credit you with suggestions. Even better, post suggestions on the popular PHP forums and spread the test infection far and wide.
Finally, if you are reading this after trying SimpleTest, then you probably already have a strong opinion of the toolkit. You can express that opinion (I won't be offended) in several places...
On the other hand if you want a particular feature added I can probably oblige or at least provide a work around. You just have to ask. If you are just having problems, feel free to contact me as well. I will usually answer support queries within a couple of days and I am building a FAQ, so all questions will be entertained. Even if you consider them silly, others will be eternally grateful.Oh yes, and thanks for reading this far.