Module::Build
Dave Rolsky is a co-author of the recently released Embedding Perl in HTML with Mason.
This article was originally published in February of 2003, and was updated by Dave Rolsky and Michael Schwern in January, 2008.
If you’ve ever created a Perl module for distribution on CPAN, you’ve used the ExtUtils::MakeMaker module. This venerable module goes back to the dawn of modern Perl, which began with the release of Perl 5.000.
Recently, Ken Williams has created a potential replacement for ExtUtils::MakeMaker called Module::Build, which was first released in August of 2002. Hugo van der Sanden, the pumpking for the current development version of Perl, has expressed interest in replacing ExtUtils::MakeMaker with Module::Build for the 5.10 release of Perl, and Michael Schwern, the current ExtUtils::MakeMaker maintainer, agrees with him. ExtUtils::MakeMaker won’t go away any time soon, but we can hope for a gradual transition to a new and improved build system.
Why ExtUtils::MakeMaker is Important
The ExtUtils::MakeMaker module, along with the h2xs script, has been a huge boon to the Perl community, as it makes it possible to have a standard way to distribute and install Perl modules. It automates many tasks that module authors would otherwise have to to implement by hand, such as turning XS into C, compiling that C code, generating man pages from POD, running a module’s test suite, and of course, installing the module.
ExtUtils::MakeMaker is a huge part of what makes PAUSE and CPAN possible, and it is quite a programming feat. Python did not have a similar utility until the September, 1999 release of distutils, PHP’s PEAR didn’t begin until mid-2000, and Ruby is just beginning to work on a standard library distribution mechanism.
The Scary Guts of ExtUtils::MakeMaker
ExtUtils::MakeMaker works by generating a makefile that contains targets for the various tasks needed when maintaining or installing a module. Here’s an example target:
install :: all pure_install doc_install
If you’re familiar with makefile syntax, you’ll realize that all this target does is call three other targets, which are named “all”, “pure_install”, and “doc_install”. These targets in turn may call other targets, or use system commands to do whatever action is needed, in this case installing the module and its associated documentation.
The makefiles generated by ExtUtils::MakeMaker are fairly complex. For example, using version 6.05 of ExtUtils::MakeMaker to install my Exception::Class module, a pure Perl distribution containing just one module, creates a makefile with about 390 lines of makefile code. Figuring out what this makefile actually does is no simple feat, because it consists of a maze of twisty macros, all alike, many of which simply call Perl one-liners from the command line to perform some task.
The ExtUtils::MakeMaker code itself is extremely complicated, as it works on many operating systems (almost as many as Perl itself), and needs to accommodate their file systems, command line shells, different versions of make, and so on. And this is all done with an extra layer of indirection in place, because it is generating a makefile which does all the work.
If you want to customize the module build or installation process, good luck. To do this, you must subclass ExtUtils::MakeMaker, override methods that generate specific makefile targets, and then tweak the generated text to include your custom behavior, all the while preserving the basic behavior of the target. Considering that there is no documentation describing what to expect from these targets, and that the actual target text may change between releases of ExtUtils::MakeMaker or between different OS’s, this can be quite painful to implement and maintain.
And by the way, you can’t really subclass ExtUtils::MakeMaker, instead you are subclassing the MY package. This is a deeply strange hack, but the end result is that you can only override certain pre-defined methods in ExtUtils::MakeMaker.
For example, the HTML::Mason module includes this snippet:
package MY;
sub test {
my $self = shift;
my $test = $self->SUPER::test(@_);
# %MY::APACHE is set in makeconfig.pl.
# If we are not going to test with Apache there is no harm in
# setting this anyway.
# The PORT env var is used by Apache::test. Don't delete it!
my $port = $MY::APACHE{port} || 8228;
$MY::APACHE{apache_dir} ||= ";
my $is_maintainer = main::is_maintainer();
# This works for older MakeMakers
$test =~ s/(runtests \@ARGV;)/\$\$ENV{MASON_VERBOSE} ==
\$(TEST_VERBOSE) ? \$(TEST_VERBOSE) : \$\$ENV{MASON_VERBOSE};
\$\$ENV{PORT}=$port;
\$\$ENV{APACHE_DIR}=q^$MY::APACHE{apache_dir}^;
\$\$ENV{MASON_MAINTAINER}=$is_maintainer; $1/;
my $bs = $^O =~ /Win32/i ? " : '\\';
# This works for newer MakeMakers (5.48_01 +)
$test =~ s/("-MExtUtils::Command::MM" "-e" ")
(test_harness\(\$\(TEST_VERBOSE\).*?\)"
\$\(TEST_FILES\))/$1 $bs\$\$ENV{MASON_VERBOSE} == \$(TEST_VERBOSE) ?
\$(TEST_VERBOSE) : $bs\$\$ENV{MASON_VERBOSE}; $bs\$\$ENV{PORT}=$port;
$bs\$\$ENV{APACHE_DIR}=q^$MY::APACHE{apache_dir}^;
$bs\$\$ENV{MASON_MAINTAINER}=$is_maintainer; $2/;
return $test;
}
The goal of all this code is to pass some additional environment information to the test scripts when they are run, so we can do tests with a live Apache server. It accommodates several versions of ExtUtils::MakeMaker, and attempts to work properly on multiple operating systems (at least Win32 and *nix), and it has to be careful about escaping things properly so that it executes properly from the shell.
Why not Perl?
All of this prompts the question of “why not just use Perl itself for all of this?” That’s exactly the question that Ken Williams answered with Module::Build. The goal of Module::Build is to do everything useful that ExtUtils::MakeMaker does, but to do this all with pure Perl wherever possible.
This greatly simplifies the build system code, and Module::Build works on systems which don’t normally include make, like Win32 and Mac OS. Of course, if a module installation requires the compilation of C code, you’ll still need an external C compiler.
Additionally, customizing Module::Build’s behavior is often quite trivial, and only requires that you know Perl, as opposed to knowing make syntax and possibly having to learn about multiple command line environments.
Module::Build also aims to improve on some of the features provided by ExtUtils::MakeMaker. One example is its prerequisite-checking system, which provides much more flexibility than what ExtUtils::MakeMaker allows. While these features could be added to ExtUtils::MakeMaker, it’s risky to make major changes to such an important module, especially one with such complex internals.
Using Module::Build
From an end-user perspective, a module that uses Module::Build looks quite a bit like one using ExtUtils::MakeMaker, and intentionally so. So to install a module using Module::Build you’d type the following lines from the command line:
perl Build.PL
./Build
./Build test
./Build install
The Build.PL script tells Module::Build to create a Build script. During this process, Module::Build also writes some files to a _build/ directory. These files are used to store the build system’s state between invocations of the Build script. This script, when invoked, simply loads up Module::Build again, and tells it to perform the specified action. An action is the Module::Build version of a makefile target, and actions are implemented in pure Perl whenever possible.
A bare bones Build.PL script might look like this:
use Module::Build;
Module::Build->new
( module_name => 'My::Module',
license => 'perl',
)->create_build_script;
The “module_name” parameter is like the ExtUtils::MakeMaker “NAME” parameter.
The “license” parameter is new with Module::Build, and its intended use it allow automated tools to determine under which license your module is distributed.
To determine your module’s version, Module::Build looks in the module specified by the “module_name” parameter, though this can be overridden either by specifying a different module to look in, or by providing the version number directly.
Of course, there are more options than those. For example, Module::Build implements a prerequisite feature similar to that implemented by ExtUtils::MakeMaker, so we can write:
Module::Build->new
( module_name => 'My::Module',
license => 'perl',
requires => { 'CGI' => 0,
'DBD::mysql' => 2.1013,
},
)->create_build_script;
If you have any experience with ExtUtils::MakeMaker, you can probably figure out that this says that our module requires any version of CGI, and version 2.1013 or greater of DBD::mysql. So far, this looks just like what ExtUtils::MakeMaker provides, but Module::Build goes further.
Consider what happens if we know that we need some piece of functionality first present in DBD::mysql 2.1013. But perhaps a release after this, 2.1014, introduced a new bug that breaks our application. If this bug is fixed in version 2.1015, we could simply require version 2.1015, but this is not ideal. There’s no reason to force someone who already has 2.1013 to upgrade because of a bug in a version they don’t have.
Fortunately, Module::Build provides a more flexible version specification option that handles exactly this situation, so we can write:
Module::Build->new
( module_name => 'My::Module',
license => 'perl',
requires => { 'CGI' => 0,
'DBD::mysql' => '>= 2.1013, != 2.1014',
},
)->create_build_script;
This says that we need a version greater than 2.1013 that is not version 2.1014. Users who have version 2.1013 or version 2.1015 or greater are not forced to upgrade, but anyone with 2.1014 will be.
If we knew that version 3.0 didn’t work with your module, we could change our specification:
Module::Build->new
( module_name => 'My::Module',
license => 'perl',
requires => { 'CGI' => 0,
'DBD::mysql' => '>= 2.1013, != 2.1014, < 3.0',
},
)->create_build_script;
If the user does have version 3.0 or greater installed, it will at least let them know that it won’t work with our module. Unfortunately, the only possible way to use our module at this point is for the end user to manually downgrade their installation of DBD::mysql, since Perl does not allow multiple versions of a module to co-exist peacefully. Still, this is better than letting the module be installed, only to fail at runtime when it tries to use an outdated API for DBD::mysql.
There are also other options related to prerequisites, such as “recommends” and “build_requires”, which can be helpful for prerequisites that are required to build the module but don’t need to be present after installation. There is also a “conflicts” option which can be used to warn a user about potential conflicts between the module they are installing and one they already have.
Action!
As of release 1.15, Module::Build implements the following actions, most of which are based on existing ExtUtils::MakeMaker functionality:
build
This is the default action, and is what happens if you run ./Build without any additional arguments. It is responsible for creating the blib/ directory and copying files into it, as well as compiling XS and C files. If you have any scripts like lib/My/Module.pm.PL, these are also run during this action.
test, testdb
Runs the module’s tests using the
Test::Harnessmodule. The “testdb” action can be used to run the tests under Perl’s debugger. Equivalently, a “debugger” parameter can be passed to the “test” action to get the same effect.clean, realclean
Both actions delete any files created by the “build” action. The “realclean” action also deletes the existing Build script.
diff
This action is used to compare the files about to be installed with any corresponding files that already exist. This feature is unique to
Module::Build.install
Installs the module files. As of version 0.15, this doesn’t yet create or install any man pages.
fakeinstall
Tells you what the “install” would do.
dist
Creates a gzip’d tarball of your distribution.
manifest Creates a MANIFEST file for your distribution.
distcheck
Tells you what files are in the build directory but not in the MANIFEST file, and vice versa.
skipcheck
Tells you what files will not be added to the MANIFEST by the “manifest” action, based on the contents of your MANIFEST.SKIP file.
distclean
This is a shortcut for “realclean” followed by “distcheck”.
distdir
Creates a directory based on your distribution name and version, and then copies all the files listed in MANIFEST to that directory. This directory is what people will see when they download and unpack your distribution.
Module::Buildalso creates a file called META.yaml which contains meta-data about your distribution. In the future, it may be possible to use a command line tool (written in Perl, of course) to read this file and use its contents to install your distribution, without running the Build.PL script. It also makes the meta-data more readily available to tools like MetaCPAN or the CPAN shell.disttest
This performs the “distdir” action, switches to the newly created directory, and then runs
perl Build.PL,./Build, and./Build test. This lets you make sure that your distribution is actually installable.help
Tells you what actions are available. If additional actions are implemented in a distribution, then these are listed here.
Any of these options can be overridden through straightforward subclassing, so our HTML::Mason example from earlier in this article might be written something like this:
package MyFancyBuilder;
use base 'Module::Build';
sub ACTION_test {
my $self = shift;
# %MY::APACHE is set in makeconfig.pl.
$ENV{PORT} = $MY::APACHE{port} || 8228;
$ENV{APACHE_DIR} = $MY::APACHE{apache_dir} || ";
$ENV{MASON_VERBOSE} ||= $self->{properties}{verbose};
# _is_maintainer_mode would be another method of our subclass
$ENV{MASON_MAINTAINER} = $self->_is_maintainer_mode();
return $self->SUPER::ACTION_test(@_);
}
This version is actually readable, and is unlikely to break regardless of changes in the Module::Build internals. This highlights just how difficult it was to accomplish a simple task using ExtUtils::MakeMaker, and how natural the pure-Perl solution can be.
The Larger Picture and Backwards Compatibility
One difficulty in getting Module::Build into widespread use is the fact that support for ExtUtils::MakeMaker is so tightly integrated into CPAN installers.
While the version of CPAN.pm which ships with 5.8 does not know how to deal with Build.PL, the latest version available from CPAN does. It will even install Module::Build for you. As of January 2008, CPANPLUS, another CPAN shell, understands Build.PL but will not install Module::Build for you, but this will be remedied in a future release.
However, old versions of CPAN.pm are still in extremely widespread use, and users won’t necessarily upgrade CPAN.pm before attempting to install a distribution that relies on Module::Build.
There are a couple workarounds for this problem. The simplest is to just include a Build.PL script and document this in the README or INSTALL files included with your distribution. This has the appeal of requiring of very little work to implement, but the downside is that people who expect things to just work with a CPAN shell will give up when your distribution doesn’t install properly.
Another possibility is to create functionally equivalent Build.PL and Makefile.PL scripts. If you’re using Module::Build because you need to customize installation behavior in a way that is difficult to do with ExtUtils::MakeMaker, this pretty much defeats the purpose of using Module::Build at all, and in any case having two separate pieces of code that do the same thing is always unappealing.
Then there’s the approach which involves using a Makefile.PL script that simply installs Module::Build if needed, and then generates a Makefile that passes everything through to the ./Build script. This is known as the “passthrough” method.
I think this approach gives the best result for the effort involved, and is the method I prefer. The Module::Build distribution includes a Module::Build::Compat module, which does the dirty work needed for this approach.
Simply add create_makefile_pl => 'passthrough' to the Build.PL parameters and a Makefile.PL will be created as part of the Build dist process.
Here’s an example of such a Makefile.PL script:
# Note: this file was auto-generated by Module::Build::Compat version 0.03
unless (eval "use Module::Build::Compat 0.02; 1" ) {
print "This module requires Module::Build to install itself.\n";
require ExtUtils::MakeMaker;
my $yn = ExtUtils::MakeMaker::prompt
(' Install Module::Build now from CPAN?', 'y');
unless ($yn =~ /^y/i) {
die " *** Cannot install without Module::Build. Exiting ...\n";
}
require Cwd;
require File::Spec;
require CPAN;
# Save this 'cause CPAN will chdir all over the place.
my $cwd = Cwd::cwd();
CPAN::Shell->install('Module::Build::Compat');
CPAN::Shell->expand("Module", "Module::Build::Compat")->uptodate
or die "Couldn't install Module::Build, giving up.\n";
chdir $cwd or die "Cannot chdir() back to $cwd: $!";
}
eval "use Module::Build::Compat 0.02; 1" or die $@;
Module::Build::Compat->run_build_pl(args => \@ARGV);
require Module::Build;
Module::Build::Compat->write_makefile(build_class => 'Module::Build');
So what exactly is going on here? A good question indeed. Let’s walk through some of the code.
unless (eval "use Module::Build::Compat 0.02; 1" ) {
print "This module requires Module::Build to install itself.\n";
require ExtUtils::MakeMaker;
my $yn = ExtUtils::MakeMaker::prompt
(' Install Module::Build now from CPAN?', 'y');
unless ($yn =~ /^y/i) {
die " *** Cannot install without Module::Build. Exiting ...\n";
}
This first attempts to load version 0.02 or greater of the Module::Build::Compat module. If it isn’t installed we know we need to install Module::Build. Because we’re polite, we ask the user if they would like to install Module::Build before going further. Some people dislike interactive installations, but fortunately the promp() command is pretty smart about detecting if there’s a user at the end of the line.
Assuming that the user agrees to install Module::Build (if they don’t the installer has to give up) this is what comes next:
# Save this 'cause CPAN will chdir all over the place.
my $cwd = Cwd::cwd();
CPAN::Shell->install('Module::Build::Compat');
CPAN::Shell->expand("Module", "Module::Build::Compat")->uptodate
or die "Couldn't install Module::Build, giving up.\n";
chdir $cwd or die "Cannot chdir() back to $cwd: $!";
We want to use CPAN.pm to actually install Module::Build, but we need to first save our current directory, because CPAN.pm calls chdir() quite a bit, and we’ll need to be in the same directory as we started in after installing Module::Build.
Then we load CPAN.pm and tell it to install Module::Build. After that, we chdir() back to our original directory.
eval "use Module::Build::Compat 0.02; 1" or die $@;
Module::Build::Compat->run_build_pl(args => \@ARGV);
require Module::Build;
Module::Build::Compat->write_makefile(build_class => 'Module::Build');
First it checks that the Module::Build install worked. Then it simply tells Module::Build::Compat to run the Build.PL script, and to write out a “passthrough” Makefile. Module::Build::Compat will attempt to convert ExtUtils::MakeMaker style arguments, like “PREFIX”, to arguments that Module::Build can understand, like “–prefix”.
The “passthrough” Makefile that Module::Build::Compat generates looks something like this:
all :
./Build
realclean :
./Build realclean
rm -f \$(THISFILE)
.DEFAULT :
./Build \$@
.PHONY : install manifest
The “.DEFAULT” target is called when there is no matching make target for the one given on the command line. It uses the “$@” make variable, which will contain the name of the target that was passed to make. So if “make install” is called, then “$@” contains “install”, and it ends up running “./Build install”.
The generated Makefile also contains a comment which specifies the module’s prerequisites, because this is how CPAN.pm figures out what a module’s prerequisites are (scary but true).
This approach is the most elegant of all, but the code that translates ExtUtils::MakeMaker arguments to something Module::Build understands is quite minimal and won’t handle all possibilities.
I have used this approach for one CPAN module, Thesaurus.pm, and in my limited testing it did work. If you are inclined to try installing this module, please send bug reports to me or the Module::Build users list, module-build-general@lists.sf.net.
Recently, Autrijus Tang submitted a more complex Makefile.PL script which implements several pieces of additional functionality. First of all, it makes sure that the script is running as a user that can actually install Module::Build. Second, it prefers CPANPLUS.pm to CPAN.pm.
Autrijus’ script looks promising, but since it hasn’t yet been tested, I’ve chosen not to include it here. It’s quite likely that some version of his script will be documented in future versions of Module::Build
Custom Behavior
As was hinted at earlier, you can directly subclass Module::Build in order to implement custom behavior. This is a big topic unto itself, and will be the topic of a future article here on perl.com.
The Future
There is plenty of work left to be done on Module::Build. Off the top of my head, here are some things that still need to be done:
The installation phase does not yet create man pages based on POD included in the distribution.
Module::Build needs to implement a “local install” feature like the one provided by the ExtUtils::MakeMaker “PREFIX” argument. The logic that implements this in ExtUtils::MakeMaker is Byzantine, but only because doing this correctly is quite complicated. This logic needs to be implemented for Module::Build as well.
Module::Build needs better backwards compatibility with ExtUtils::MakeMaker. The argument translation in Module::Build::Compat is currently just a placeholder. Things like “PREFIX”, “LIB”, and “UNINST=1” all need to be translated by Module::Build::Compat, and the equivalent functionality needs to be implemented for Module::Build
CPANPLUS.pm could take advantage of more Module::Build features. For example, it currently ignores the “conflict” information that Module::Build makes available, and it doesn’t attempt to distinguish between “build_requires”, “requires”, or “recommends”.
Some of what Module::Build provides is intended for use by external tools, such as the meta-data provided by the META.yaml file. CPANPLUS.pm could use this to avoid having to run the Build.PL and Build scripts, thus avoiding the need to install any of the “build_requires” modules. Package managers like rpm or the Debian tools could also use it to construct installable packages for Perl modules more easily.
Adding at least basic support for Module::Build to CPAN.pm would be nice. If anyone is interested in pursuing this, I have an old patch that may provide a decent start on this. Contact me if you’re interested.
More Information
If you want to learn more about Module::Build, the first thing you should do is install it (it will install itself just fine under CPAN.pm) and read the docs for the Module::Build and Module::Build::Compat modules. The source is on GitHub.
Thanks
Thanks to Ken Williams for reviewing this article before publication, and for writing Module::Build.
Tags
Feedback
Something wrong with this article? Help us out by opening an issue or pull request on GitHub
