perl.com: Becoming a CPAN Tester with CPANPLUS

Becoming a CPAN Tester with CPANPLUS
by Autrijus Tang | Pages: 1, 2

Reporting Test Results

Despite the handy utility of the c command, there will be no reports if nobody submits them. CPANPLUS allows you to send a report whenever you've done a make test in the course of installing a module; to enable this feature, please turn on the cpantest configuration variable:

 CPAN Terminal> s cpantest 1
 CPAN Terminal> s save
 Your CPAN++ configuration info has been saved!

Afterward, just use CPANPLUS as usual. You will be prompted during an installation, like below:

 CPAN Terminal> i DBD::SQLite
 Installing: DBD::SQLite
 # ...
 t/30insertfetch....ok
 t/40bindparam......FAILED test 25
     Failed 1/28 tests, 96.43% okay
 t/40blobs..........dubious
     Test returned status 2 (wstat 512, 0x200)
     DIED.  FAILED tests 8-11
     Failed 4/11 tests, 63.64% okay
 Failed 2/19 test scripts, 89.47% okay.
 5/250 subtests failed, 98.00% okay.
 *** Error code 9
 Report DBD-SQLite-0.15's testing result (FAIL)? [y/N]: y

It always defaults to n, so you won't send out bogus reports by mistake. If you enter y, then different actions will happen, depending on the test's outcome:

For modules that passed their tests, a pass report is sent out immediately to the cpan-testers list.
For modules that didn't define a test, an unknown report is sent to the author and cpan-testers; the report will include a simple test script, encourage the module author to include it in the next release, and contain a URL of Michael Schwern's Test::Tutorial manpage.
If the module fails at any point (perl Makefile.PL, make, or make test), then the default editor is launched to edit its fail report, which includes the point of failure and the captured error buffer.
You are free to add any text below the Additional comments line; to cancel a report, just delete everything and save an empty file.

Before sending out a fail report, be sure to double-check if it is really the module's problem. For example, if there's no libgtk on your system, then please don't send a report about failing to install Gtk.

Also, be prepared to follow up with additional information when asked. The author may ask you to apply some patches, or to try out an experimental release; do help the author whenever you can.

Batch Testing and cpansmoke

While regular users can test modules as they install via the cpantest option, and often provide important first-hand troubleshooting information, we still need dedicated testers -- they can broaden the coverage of available platforms, and may uncover problems previously unforeseen by the author. Dedicated testing and regular reports are complementary to each other.

Historically, CPAN testers would watch for notices of recent PAUSE uploads posted on the cpan-testers list, grab them, test them manually, and run the cpantest script, as we've seen in What is CPAN Testers?. That script is bundled with CPANPLUS; please refer to its POD documentation for additional options.

However, normally you won't be using it directly, as CPANPLUS offers the cpansmoke wrapper, which consolidates the download => test => copy buffer => run cpantest => file report procedure into a single command:

 % cpansmoke Mail::Audit Mail::SpamAssassin          # by module
 % cpansmoke Mail-Audit-2.1 Mail-SpamAssassin-2.20   # by distname
 % cpansmoke /S/SI/SIMON/Mail-Audit-2.1.tar.gz       # or full name

Due to the need of testing new distributions as they are uploaded, cpansmoke will use http://www.cpan.org/ as its primary host, instead of the local mirror configured in your CPANPLUS settings. This behavior may be overridden with the -l flag.

Since cpansmoke will stop right after make test rather than installing any modules, it can test modules with different versions than the ones installed on the system. However, since it won't resolve prerequisites by default, you'll need to specify the -p flag to test and report them.

The -d flag will display previous results before testing each module; similarly, -s will skip testing altogether if the module was already tested on the same platform. The latter should only be used with automated test (to reduce spamming), since the same platform can have different results.

There are numerous other flags available; please consult cpansmoke's manpage for further information.

Automatic Testing

Besides interactive batch-testing, cpansmoke also provides the capability of unattended testing; this concept is also covered by Mozilla Project's Tinderbox toolkit, as explained in its homepage:

 There is a flock of dedicated build machines that do
 nothing but continually check out the source tree and
 build it, over and over again.

Essentially, if you have a machine with plenty of hard disk space and adequate bandwidth, you can set up cpansmoke so it tests each new module as it is uploaded.

The -a option lets cpansmoke enter the non-interactive mode. In this mode, only failures during make test are reported, because errors that occured during Makefile.PL or make are more likely the machine's problem, not the module's.

Additionally, you should specify the -s flag to avoid duplicate reports, and -p to avoid false-negatives caused by unsatisfied dependencies.

Setting up an auto-smoking machine is simple, using a mail filtering mechanism; just join the CPAN Testers list by sending an empty email to cpan-testers-subscribe@perl.org, and add the lines below to your Mail::Audit filter:

 fork || exec("sleep 40000 && cpansmoke -aps $1 >/dev/null 2>&1")
     if $mail->subject =~ /^CPAN Upload: (.*)$/;

This rule will fork out a new cpansmoke process for each incoming mail about a CPAN upload. Note that it works only if incoming mail arrives no less than once every hour; otherwise it might fork-bomb your system to a grinding halt.

In case you're wondering, the sleep 40000 line is due to the fact that a PAUSE upload needs several hours to appear on www.cpan.org. Michael Schwern suggested that using a smoke queue would be a better solution, since it allows error-checking and logging; implementations are welcome.

If you are stuck with procmail, then here's the recipe that does the same (but do consider upgrading to Mail::Audit):

 :0hc
 * ^Subject: CPAN Upload:
 |sh -c "sleep 40000 && grep Subject|cut -f4 -d' '|xargs cpansmoke -aps >/dev/null 2>&1"

If you have suggestions on how to make this work on non-unix systems, please do let me know.

Testing-Related APIs

A programmable interface is the primary strength of CPANPLUS. All features listed above are available as separate methods of the CPANPLUS::Backend object. Since the module's manpage already contains detailed information, I'll just list some sample snippets below.

To show all reports of Acme::* on FreeBSD:

 use CPANPLUS::Backend;
 my $cp = CPANPLUS::Backend->new;
 my $reports = $cp->reports( modules => [ map $_->{package}, values(
     %{ $cp->search(type => 'module', list => ['^Acme::\b']) }
 ) ] );

 while ( my ($name, $href) = each (%$reports) ) {
     next unless $href;
     my @entries = grep { $_->{platform} =~ /\bfreebsd\b/ } @$href;
     next unless @entries;

     print "[$name]\n";
     for my $rv (@entries) {
         printf "%8s %s%s\n", @{$rv}{'grade', 'platform'},
                              ($rv->{detail} ? ' (*)' : '');
     }
 }

To test all Acme::* modules, but install all needed prerequisites:

 use CPANPLUS::Backend;
 my $cp = CPANPLUS::Backend->new;
 $cp->configure_object->set_conf( cpantest => 1 );
 $cp->install(
     modules => [ map $_->{package}, values(
         %{ $cp->search(type => 'module', list => ['^Acme::\b']) }
     ) ], 
     target        => 'test',
     prereq_target => 'install',
 );

If you need to fine-tune various aspects during testing (timeout, prerequisite handling, etc.), then please consult the source code of cpansmoke; most tricks involved are documented in comments.

Unfortunately, the process of editing and sending out reports remains as the only non-programmable part, since the cpantest script doesn't exist as a module. Although Skud has a CPAN::Test::Reporter on CPAN, its format is sadly out-of-sync with cpantest, and may generate invalid reports (as of version 0.02). Any offers to back-port the cpantest script into that module would be greatly appreciated.

Conclusion

As purl said, ``CPAN is a cultural mechanism.'' It is not an abstraction filled with code, but rather depends on people caring enough to share code, as well as sharing useful feedback in order to improve each other's code.

With the advent of bug-tracking service and automatic testing, the social collaboration aspect of CPAN is greatly extended, and could be developed further into a full-featured Tinderbox system, or linked with each module's source-control repositories.

Also, since CPANPLUS is designed to accommodate different package formats and distribution systems, it provides a solid foundation for projects like Module::Build (make-less installs), NAPC (distributed CPAN) and ExtUtils::AutoInstall (feature-based dependency probing)... the possibilities are staggering, and you certainly won't be disappointed in finding an interesting project to work on. Happy hacking!

Acknowledgements

Although I have done some modest work in integrating CPANPLUS with CPAN testing, the work is really built on contributions from several brilliant individuals:

First, I'd like to dedicate this article to Elaine -HFB- Ashton, for her tireless efforts on Perl advocacy, and for sponsoring my works as described in this article.

Thanks also goes to Graham Barr and Chris Nandor, for establishing the CPAN Testers at the first place, and coming up with two other important cultural mechanisms: CPAN Search and Use Perl; respectively.

To Jos Boumans, Joshua Boschert, Ann Barcomb and everybody in the CPANPLUS team -- you guys rock! Special thanks for another team member, Michael Schwern, for reminding everybody constantly that Kwalitee Is Job One.

To Paul Schinder, the greatest tester of all time, who submitted 10551 reports by hand out of 23203 to date, and kept CPAN Testers alive for a long time. And thanks for every fellow CPAN Tester who have committed their valuable time -- I hope cpansmoke can shoulder your burden a little bit.

To Simon Cozens, for his swiss-nuke Mail::Audit module, and for keep asking me to write 'something neat' for perl.com. :-)

To Jarkko Hietaniemi, for establishing CPAN; and to Andreas J. Koenig, for maintaining PAUSE and showing us what is possible with CPAN.pm.

Finally, if you decide to follow the steps in this article and participate in the testing process, then you have my utmost gratitude; let's make the world a better place.