An Introduction to Testing

by chromatic
December 04, 2001

Someday, you'll be dubiously blessed with the job of maintenance programming. You might need to add new features or to fix long-standing bugs. The code may be your own or the apparently disturbed mutterings of a long-disappeared agent of chaos. If you haven't yet been this fortunate, then download a Perl CGI script circa 1996 and try to make it operate under use strict, warnings and taint mode.

Maintenance is rarely pretty, mixing forensics, psychology and playing-card house construction. Complicating the matter are concerns such as backward compatibility, portability and interoperability. You'll probably hate the learning experience, but don't miss the chance to learn some important lessons:

• Software spends more time being maintained than being developed.
• Things that were obvious in development are easily obscured.
• Fixing bugs is easier than finding them and making sure nothing else breaks.
  
• The real cost of software is programmer time and effort, not length of code, licenses or required hardware.

These rules have been well-established since the early days (think Grace Hopper) of software engineering. Good practices remain the same: Write good comments, document your assumptions and logic, test your code to death, test it again.

Learning Perl By Randal L. Schwartz & Tom Phoenix Table of Contents Index Sample Chapter Read Online -- Safari

Every serious software engineering methodology promotes testing. It is essential to newer approaches, like Extreme Programming. A comprehensive test suite helps to verify that the code performs as expected, helps to minimize the scope of future modifications, and frees developers to improve the structure and design of code without changing its behavior. Yes, this can actually work.

Caveats

Experienced readers (especially those with strong math backgrounds) rightly note that testing cannot prove the absence of bugs. If it's theoretically impossible to write a non-trivial program with zero defects, it's impossible to write enough tests to prove that the program works completely. Programmers try to write bug-free code anyway, so why not test anything and everything possible? A tested program may have unknown bugs, but an untested program will.

Of course, some things are truly untestable. This category includes the ``black boxes,'' such as other processes and machines, and system libraries. More things are testable than not, though. Perl allows good programmers to perform scary black magic, including system table and run-time manipulations. It's possible to create your own fake little world just to satisfy a testable interface. A little megalomania can be handy.

Testing can be difficult. A wad of code with minimal documentation -- a design from the Perl 4 days -- and business logic that relies on prayer, global variables and animal sacrifice may push you to new heights of productivity, or teach you how to manage programmers. If you don't fix it now, then when do you fix it? Extreme Programming recommends revising untestable code to make it easier to maintain and to test. Sometimes, you must write simple tests, rework the code slightly, and iterate until it's palatable.

Don't let the enormity of the task get you down. If you can write Perl code worth testing, then you can write tests.

How Perl Module Testing Works

This section assumes you're already familiar with perlmodinstall, having installed a module manually. After the perl Makefile.PL and make stages, make test runs any tests shipped with the module, either test.pl or all files in the t/ directory that end in ``.t.'' The blib/ directories are added to @INC, and Test::Harness runs the test files, captures their output and provides a short summary of the results, including success and failure.

At its heart, a test either prints ``ok'' or ``not ok.'' That's it. Any test program or testing framework that prints results to standard output can use Test::Harness. If you're feeling epistemological (a good trait to cultivate for writing tests), you could ask ``What is truth?'':

        print "1..1\n";

        if (1) {
                print "ok\n";
        } else {
                print "not ok\n";
        }

Basic? Yes. Bogus? Not really. This is a variant of an actual Perl core test. If you understood that code, then you can write tests. Ignoring the first line for now, simply stick it in a file (truth.t) and run either the command line:

        perl -MTest::Harness -e "runtests 'truth.t'";

or the program:

        #!/usr/bin/perl -w

        use strict;
        use Test::Harness;

        runtests 'truth.t';

This should produce a message saying that all tests succeeded. If not, then something is broken, and many people would be interested in fixing it.

The first line of the test corresponds to Test::Harness' handy test-numbering feature. The harness needs to know how many tests to expect, and each individual test within a group can have its own number. This is for your benefit. If one test in a 100 mysteriously fails, then it's much easier to track down number 93 than it is to run through the debugger, comment out large swaths of the test suite or rely on intuitively placed print statements.

Knowing truth is good, and so is discerning falsehood. Let's extend truth.t slightly. Note the addition of test numbers to each potential printable line. This is both a boon and a bane.

        print "1..2\n";
		
        if (1) {
                print "ok 1\n";
        } else {
                print "not ok 1\n";
        }

        if (0) {
                print "not ok 2\n";
        } else {
                print "ok 2\n";
        }

Besides the increasingly duplicate code, keeping test numbers synchronized is painful. False laziness is painful -- Test::Harness emits warnings if the number of actual tests run does not meet its expectations. Test writers may not mind, but spurious warnings will confuse end users and healthily lazy developers. As a rule, the simpler the output, the more people will believe that things succeeded. The stuffed, smiling Pikachu (hey, it was a birthday present from an attractive female Web designer) perched atop my monitor makes me think that a giant yellow smiley face would be even better than a simple ``ok'' message. ASCII artists, fire up your editors!

Unfortunately, the truth test is repetitive and fragile. Adding a third test between the first two (the progression from ``truth'' to ``hidden truth'' to ``falsehood'' makes sense) means duplicating the if/else block and renumbering the previously second test. There's also room for a subtle bug:

        print "1..2\n";

        if (1) {
                print "ok 1\n";
        } else {
                print "not ok 1\n";
        }

        if ('0 but true') {
                print "ok 2\n";
        } else {
                print "not ok 2\n";
        }

        if (0) {
                print "not ok 3\n";
        } else {
                print "ok 3\n";
        }

Forgetting to update the first line is common. Two tests were expected; three tests ran. The confused Test::Harness will report strange things, like negative failure percentages. Baby Pikachu may cry. Smarter programmers eventually apply good programming style, writing their own ok() functions:

        print "1..3\n";

        my $testnum = 1;
        sub ok {
                my $condition = shift;
                print $condition ? "ok $testnum\n" : "not ok $testnum\n";
                $testnum++;
        }

        ok( 1 );
        ok( '0 but true' );
        ok( ! 0 );

The lowest levels of the Perl core test suite use this approach. It's simpler to write and handles numbering almost automatically. It lacks some features, though, and is little easier to debug.

Pages: 1, 2

Next Page