Becoming a CPAN Tester with CPANPLUS
by Autrijus TangApril 30, 2002
"This is a great war long-planned,
and we are but one piece in it,
whatever pride may say.
... And now, all realms shall be put to the test."
-- Beregond, <<Lord of the Rings>>
Introduction
In CPANPLUS -- Like CPAN.pm only better?, Jos Boumans recounted the tale of how his CPANPLUS project came to be; one of the original goals was to provide a modular backend to CPANTS, the CPAN Testing Service, so modules can be automatically built and tested on multiple platforms.
Although the initial CPANPLUS release did not support testing-related features, Jos considered them important enough to be listed near the top of his Current and Future Developments wish list, and quoted Michael Schwern's opinion on their potential benefits*:
It would alter the way free software is distributed. It would get a mark of quality; it would be tested and reviewed. And gosh darnit, it would be good.
At that time, despite having participated in CPANPLUS's development for more than four months, I was blissfully unaware of the project's original vision. But it seemed like a worthwhile goal to pursue, so I quickly jotted down several pieces that need to work together:
- Check test reports of modules before installation.
-
Report
make testresults to the author, and make them available for other users to check. - Tools to batch-test modules with or without human intervention.
- A clean API for other programs to perform tests, reporting and queries.
Today, with the 0.032 release of CPANPLUS, all the above pieces are completed. This article will show you how to configure and use these tools, as well as describing some important new features in CPANPLUS that made them possible.
Setting Up the Environment
First, you need a copy of CPANPLUS 0.032 or above. I recommend the tarball automatically built from the distribution branch, available at http://egb.elixus.org/cpanplus-dist.tar.gz; it should contain the latest bug-fixes and features marked as stable.
Since Jos' article and README already contain install instructions, I will not repeat the details. However, please note that the dependency detection logic in Makefile.PL has been changed since Jos' previous article, which means when faced with this prompt:
[CPAN Test reporting] - Mail::Send ...failed! (needs 0.01) - File::Temp ...loaded. (0.13 >= 0.01) ==> Do you wish to install the 1 optional module(s)? [n]
You should generally answer y here, as only modules that can safely
install on your machine are displayed. The default n does not mean
we don't recommend it; it merely means it isn't mandatory for the
bare-bone CPANPLUS to function.
For this article's purpose, you need to answer y to all such
questions, since we will be using LWP, Mail::Send and IPC::Run
extensively in testing-related functions.
After running Makefile.PL, you should run make test* like every
good user; if any dependent modules were selected in the previous step,
then you will have to run make test as root, so it can fetch and install
these modules automatically before testing itself.
After the comforting All tests successful message appears on your
screen, just do a make install, and we are ready to go.
Why Is Testing Important?
... or not. What if, instead of All tests successful, you see a
series of cryptic messages:
% sudo make test /usr/bin/perl Makefile.PL --config=-target,skiptest --installdeps=Term::Size,0.01 *** Installing dependencies... *** Installing Term::Size... Can't bless nonreference value at lib/CPANPLUS/Internals/Module.pm line 239. Compilation failed in require at Makefile.PL line 20. make: *** [installdeps] Error 255
An experienced Perl user will probably take the following steps:
- Look at lib/CPANPLUS/Internals/Module.pm and see whether it's a trivial mistake. Unfortunately, it is not.
- Search README for the project's mailing list address. Discovering that it's hosted in SourceForge, check Geocrawler (or any search engine) for existing bug reports and discussions. Unfortunately, there are none.
- Copy-and-paste the error message into a bug-reporting e-mail to the mailing list, including your operating system name and version, Perl version, and wait for a fix.
Apparently, the above is exactly what Juerd Waalboer (the bug's original reporter) did when he ran into this trouble. Thanks, Juerd!
However, there are a number of problems with the procedure above. It might be thwarted at every step:
- What if Juerd had been less experienced, and unaware of this bug-reporting procedure?
- What if he decided to kluge around it by commenting out line 239, instead of taking the laborious copy-and-paste path to report it?
- CPANPLUS has a mailing list; other modules might not. Worse, they may not even have a working contact address -- maybe the author is on vacation.
-
He might not have the sense to provide adequate debug-related data,
and simply wrote
CPANPLUS didn't work! It can't install!#!@#@#!in red-colored blinking HTML email, wasting bandwidth and causing developers to ignore his mail completely. - Even after a fix was made available, and posted as a patch on the mailing list, other people about to install CPANPLUS 0.03 still have no way to find out about it beforehand. Thus, when the same problem occurs again, they may forget to search in the archives, resulting in duplicate bug reports, or (worse) more dissatisfied users.
-
Actually, we had not mentioned the very likely scenario: What if
Juerd didn't run
make testat all? If so, the developers will have no way to know that this bug has ever happened. - Finally, while CPANPLUS (like most CPAN modules) includes a regression test suite, some modules may omit their tests altogether. Users may not feel comfortable to mail the author to request a test script to accompany the distribution; consequently, bugs will surface in harder-to-detect ways, and may never get fixed.
As you can see, both authors and users can certainly benefit a lot from an improved test reporting system -- one that simplifies the process of reporting a failed (or successful) installation, and allows users to check for existing reports before downloading a module.
Such a system already exists; it's called CPAN Testers.
What Is CPAN Testers?
When most people run make test, it's immediately followed by a
make install; that is, people only test modules that they are going
to use. But they may not have the time or inclination to report the
outcome, for various reasons listed above.
CPAN Testers (http://testers.cpan.org/) is an effort to set up a Quality Assurance (QA) team for CPAN modules. As its homepage states:
The objective of the group is to test as many of the distributions on CPAN as possible, on as many platforms as possible.
The ultimate goal is to improve the portability of the distributions on CPAN, and provide good feedback to the authors.
Central to its operation is a mailing list, cpan-testers@perl.org, and a small program, cpantest. The program is a convenient way to report a distribution's test result, which is sent to the mailing list using an uniform format. For example:
# test passed, send a report automatically (no comments) cpantest -g pass -auto -p CPANPLUS-0.032 # test failed, launch an editor to edit the report, and cc to kane cpantest -g fail -p CPANPLUS-0.032 kane@cpan.org
The argument (pass/fail) after the -g option is called the
grade -- the overall result of a test run. It may also be na
(not available), which means the module is not expected to work on this
platform at all (Win32::Process on FreeBSD, for example), and
unknown in the case that no tests are provided in the distribution.
All recent CPAN uploads, as well as all test results, are forwarded
to the cpan-testers mailing list's subscribers -- CPAN testers.
After a distribution's release on CPAN, one or more testers will
pick it up, test it, then feed the result back to the author and to
the list. The testers don't have to know what the module is about;
all they do is to ensure it's working as advertised.
Test reports on the mailing list are automatically recorded in a ever-growing database. http://testers.cpan.org/search is its search interface, where you can query by distribution name, version, or the testing platform.
Theoretically, you can query that database before installing any CPAN module, to see whether it works on your platform, and check for associated bug reports. The same information is also present at each module's page in http://search.cpan.org/, and maybe http://rt.cpan.org/ in the near future.
Alas, while this system certainly works, it is far from perfect. Here are some of its shortcomings:
- The integration between Web site and mailing list hadn't extended to the CPAN.pm shell, so checking for test results requires an additional step to navigate the Web browser.
-
People who aren't subscribed to the
cpan-testerslist rarely submit test reports, so coverage of platforms is limited to a few popular ones; a heavily used module may still be tested in only two or three platforms, hardly representing its entire user base. - There are no smoking (automatic cross-platform testing) mechanisms; all testers must manually fetch, extract, test and submit their test reports, including the same copy-and-paste toil we described earlier. This entry barrier has seriously limited the number of active volunteer testers.
Fortunately, CPANPLUS addressed most of these issues, and made it significantly easier to participate in testing-related activities. Let's walk through them in the following sections.
Checking test reports
CPANPLUS offers a straightforward way to check existing reports.
Simply enter cpanp -c ModuleName in the command line:
% cpanp -c CPANPLUS [/K/KA/KANE/CPANPLUS-0.032.tar.gz] PASS freebsd 4.5-release i386-freebsd
As you can see, this reports the most recent version of CPANPLUS, which passed its tests on FreeBSD as of this writing.
You can also specify a particular version of some distribution. For example, the version Juerd was having problems with is 0.03, so he can do this:
% cpanp -c CPANPLUS-0.03
[/K/KA/KANE/CPANPLUS-0.03.tar.gz]
FAIL freebsd 4.2-stable i386-freebsd (*)
PASS freebsd 4.5-release i386-freebsd (*)
PASS linux 2.2.14-5.0 i686-linux
==> http://testers.cpan.org/search?request=dist&dist=CPANPLUS
As you can see, there are three reports, two of which contain
additional details (marked with *), available at the URL listed
above. The failed one's says:
This bug seems to be present in machines upgrading to 0.03 from 0.01/0.02 releases. (It has since been fixed in the upcoming 0.031 release.)
Which exactly addressed Juerd's original problem.
Another useful trick is using the o command in the CPANPLUS
Shell to list newer versions of installed modules on CPAN, and
check on them all with c *:
% cpanp CPAN Terminal> o 1 0.14 0.15 DBD::SQLite MSERGEANT 2 2.1011 2.1014 DBD::mysql JWIED
CPAN Terminal> c *
[/M/MS/MSERGEANT/DBD-SQLite-0.15.tar.gz]
FAIL freebsd 4.5-release i386-freebsd (*)
PASS linux 2.2.14-5.0 i686-linux
PASS solaris 2.8 sun4-solaris
==> http://testers.cpan.org/search?request=dist&dist=DBD-SQLite
[/J/JW/JWIED/DBD-mysql-2.1014.tar.gz]
FAIL freebsd 4.5-release i386-freebsd (*)
PASS freebsd 4.5-release i386-freebsd
==> http://testers.cpan.org/search?request=dist&dist=DBD-mysql
This way, you can safely upgrade your modules, confident in the knowledge that the newer version won't break the system.
Pages: 1, 2 |
