Testing Files and Test Modules
by Phil Crow
|
Pages: 1, 2, 3
Testing a Test Module
The most interesting part of writing Test::Files was learning how to test it. Thanks to Schwern, I learned about Test::Builder::Tester, which eases the problems inherent in testing a Perl test module.
The difficulty with testing Perl tests has to do with how they normally run. The venerable test harness scheme expects test scripts to produce pass and fail data on standard out and diagnostic help on standard error. This is a great design. The simplicity is exactly what you would expect from a Unix-inspired tool. Yet, it poses a problem for testing test modules.
When eventual users use the test module, their harness expects it to write quite specific things to standard out and standard error. Among the things that must go to standard out are a sequence of lines such as ok 1. When you write a test of the test module, its harness also expects to see this sort of data on standard out and standard error. Having two different sources of ok 1 is highly confusing, not least to the harness, which chokes on such duplications.
Test module writers need a scheme to trap the output from the module being tested, check it for correct content, and report that result onto the actual standard channels for the harness to see. This is tricky, requiring care in diversion of file handles at the right moments without the knowledge of the module whose output is diverted. Doing this by hand is inelegant and prone to error. Further, multiple test scripts might have to recreate home-rolled solutions (introducing the oldest of known coding sins: duplication of code). Finally, the diagnostic output, in the event of failure, from homemade diverters is unlikely to be helpful when tests of the test module fail.
Enter Test::Builder::Tester.
To help us test testers, Mark Fowler collected some code from Schwern, and used it to make Test::Builder::Tester. With it, tests of test modules are relatively painless and their failure diagnostics are highly informative. Here are two examples from the Test::Files test suite. The first shows a file comparison that should pass:
test_out("ok 1 - passing file");
compare_ok("t/ok_pass.dat", "t/ok_pass.same.dat",
"passing file");
test_test("passing file");This test should work, generating ok 1 - passing file on standard output. To tell Test::Builder::Tester what the standard output should be, I called test_out. After the test, I called test_test with only the name of my test. (To avoid confusion, I made the test names the same.)
Between the call to test_out and the one to test_test, Test::Builder::Tester diverted the regular output channels so the harness won't see them.
The second example shows a failed test and how to check both standard out and standard error. The later contains the diagnostic data the module should generate.
test_out("not ok 1 - failing file");
$line = line_num(+9);
test_diag(" Failed test (t/03compare_ok.t at line $line)",
'+---+--------------------+-------------------+',
'| |Got |Expected |',
'| Ln| | |',
'+---+--------------------+-------------------+',
'| 1|This file |This file |',
'* 2|is for 03ok_pass.t |is for many tests *',
'+---+--------------------+-------------------+' );
compare_ok("t/ok_pass.dat", "t/ok_pass.diff.dat",
"failing file");
test_test("failing file");Two new functions appear here. First, line_num returns the current line number plus or minus an offset. Because failing tests report the line number of the failure, checking standard error for an exact match requires matching that number. Yet, no one wants his tests to break because he inserted a new line at the top of the script. With line_num, you can obtain the line number of the test relative to where you are. Here, there are nine lines between the call to line_num and the actual test.
The other new function is test_diag. It allows you to check the standard error output, where diagnostic messages appear. The easiest way to use it is to provide each line of output as a separate parameter.
Summary
Now you know how to use Test::Files and how to test modules that implement tests. There is one final way I use Test::Files. I use it outside of module testing any time I want to know how the contents of text files in two directory hierarchies compare. With this, I can quickly locate differences in archives, for example, enabling me to debug builders of those archives. In one example, I used it compare more than 400 text files in two WebSphere .ear archives. My program had only about 30 operative lines (there were also comments and blank lines) and performed the comparison in under five seconds. This is testament to the leverage of Perl and CPAN.
(Since doing that comparison, I have moved to a new company. In the process I exchanged WebSphere for mod_perl and am generally happier with the latter.)
You must be logged in to the O'Reilly Network to post a talkback.
