We've expanded our Perl news coverage and improved our search! Search for all things Perl across O'Reilly!

Building Testing Libraries

by Casey West
May 07, 2004

Testing is an important step in developing any important body of work. In today's pragmatic culture, we're taught to test first, test often, and design with tests. The expectation is that chanting "test test test" forgives all sins. To a large extent, this is true. Testing helps us produce quality software at all scales.

The extreme code produced by this extreme lifestyle hides in the test suite itself. Often the ugliest code we write resides in files with a .t extension. Riddled with redundant, ghastly expressions, the test suite is the collateral damage on our road to beautiful production code.

Let's review some common pitfalls made when testing. Many of these testing procedures may be new to you. Serious headway has been made in recent history with the testing libraries on the CPAN.

A Test File is Just a Program

Each test file is a program, just as important as any other program you'd write that uses software being tested. It must be treated with the same care. If you plan to use strict and warnings in a program related to the code you're testing, be sure to do the same in your tests.

Each test file should start with these three lines.

  #!/path/to/perl
  use strict;
  use warnings;

If you plan to run your software in a taint-checked environment, which is considered a good idea, then supply the -T command-line option to the #! line.

  #!/path/to/perl -T

This will ensure that you won't make syntactic mistakes in your test files. It will also require your software to work correctly in a restricted environment.

Be Compatible with Test::Harness

Related Reading Extreme Programming Pocket Guide By chromatic

Test::Harness is a very useful Perl module for running test suites. If you are building a Perl module yourself, and using ExtUtils::MakeMaker or Module::Build for the build process, you're using Test::Harness. If you aren't using any of these mechanisms, do try to be compatible with it. This will help other users and developers of your software who are used to dealing with Test::Harness.

Compatibility comes in the form of the test file's output. Test::Harness will run your program and record its output to STDOUT. Anything sent to STDERR is ignored, silently passed on to the user. There are particulars about testing under the harness that should be observed. The basics are simple.

When a test passes, it outputs a line containing ok $N, where $N is the test number. When a test fails, the line contains not ok $N. Test numbers are optional but recommended. Tests may be named. Anything after the number, $N, is considered the test name, up to a hash (#). Anything following the hash is a comment.

Furthermore, you are encouraged to supply a header. The header tells Test::Harness how many tests you expect to run and should be the first thing you output. If you're unsure of the number of tests, the header may be the very last thing output. Its format is also simple: 1..$M, where $M is the total number of tests to run. The header helps the harness figure out how well your tests did.

Any other output should be commented on lines beginning with a hash (#). Here is an example of prototypical output understood by Test::Harness.

  1..4
  ok 1 - use Software::Module
  ok 2 - object isa Software::Module
  not ok 3 - $object->true() should return true
  #     Failed test (test.t)
  #          got: undef
  #     expected: 1
  ok 4 # skip Net::DNS required for this test
  # Looks like you failed 1 tests of 4.

Use a Testing Module

A simple way to achieve Test::Harness compatibility is to use a testing module from the CPAN. Many test suites over the years have reinvented the ok() function, for example.

  {
    my $N = 1;
    sub ok($;$) {
        my ($test, $name) = @_;
        print "not " unless $test;
        print "ok $N - $name\n";
        $N++;
    }
  }

There is no need to do this, however. The standard Perl distribution comes with testing modules. Two great options are Test::Simple and Test::More. Test::Simple is a great way to get your feet wet; it implements only the ok() function. Test::More has more features and is recommended when you write your test suites.

Using Test::More is very simple; many have written on the subject. This is how you would achieve the output described in the previous section.

  #!/usr/bin/perl -T
  use strict;
  use warnings;
  use Test::More tests => 4;
  
  use_ok 'Software::Module';
  my $object = Software::Module->new;
  isa_ok $object, 'Software::Module', 'object';
  cmp_ok $object->true, 1, '$object->true() should return true';
  
  SKIP: {
      skip 1, "Net::DNS required for this test"
        unless eval 'require Net::DNS';
      
      ok $object->network(), "run over network";
  }

Don't Iterate, Compare

I've often seen tests that loop over a list and check each item to be sure the list is correct. While this approach makes you feel good, artificially adding to the number of tests you've written, it can be sloppy and long-winded. Here is an example.

  my @fruits = qw[apples oranges grapes];
  my @result = get_fruits();
  foreach my $n ( 0 .. $#fruits ) {
      is $result[$n], $fruits[$n], "$fruits[$n] in slot $n";
  }
  is scalar(@result), scalar(@fruits), "fruits the same size";

It looks like four tests were written; the reality is that one test was written poorly. Test::More has several utility functions to get the job done. In this test, @fruits represents a set of non-repeatable fruits I expect to get back from get_fruits(). As such, I can use eq_set() to test this function in one quick try.

  my @fruits = qw[apples oranges grapes];
  my @result = get_fruits();
  ok eq_set(\@result, \@fruits), "got [@fruits]";

That was easy and short. But what happens when you have a deep data structure that you're dying to test? That's where Test::Deep comes in. Downloadable from the CPAN, this module provides the cmp_deeply() function. Here is a simple example.

  use Test::Deep;
  my $people = [
    {
      name     => "Casey West",
      employer => "pair Networks",
    },
    {
      name     => "Larry Wall",
      employer => "The Perl Foundation",
    },
  ];
  
  my $result = $dude->contacts->retrieve_all;
  
  cmp_deeply $result, $people, 'contacts match';

This example scratched the surface of what Test::Deep is capable of. When you've got to test a complex data structure, especially in a complex way, use this module. Here is a more difficult example made testable by this module. In this example, $dude->contacts->retrieve_all returns an unordered list of contacts with various bits of information associated with each of them.

  use Test::Deep;
  my $person = {
    name     => re("^[\w\s]+$"),
    employer => ignore(),
    age      => code(sub { shift > 18 }),
  };
  my $people = array_each($person);
  my $result = $dude->contacts->retrieve_all;
  
  cmp_deeply $result, $people, 'contacts match';

This code, using only functions exported by Test::Deep, does a lot of work. Each person has a definition that should match $person. Every person in the $result list is a hash reference containing three elements. name must match the regular expression /^[\w\s]+$/, employer must exist and its value is ignored, and age should be over 18 or it will fail. array_each() returns an object that instructs cmp_deeply that every value in a list must match the definition provided.

This small amount of code accomplishes quite a lot. Test::Deep has saved us from wasting time and working hard to solve a difficult problem. It has made the hard things possible.

Pages: 1, 2

Next Page