Articles on Perl.com - programming news, code and culture

Bang Bang

Wed, 11 Nov 2020 11:04:40 +0000

Interpreters read and execute scripts (whereas shells are more like a kitchen pass-through and can either execute or hand over to another interpreter). When we specify interpreter on the command line, it is the one that will be used. For instance Rscript script.R will execute script.R using the Rscript interpreter.

When we execute a file without explicitly giving an interpreter (for instance, like ./myscript.pl), it is the job of the “shebang” to tell to the shell/OS which interpreter to use. The shebang is that first line of a text file that starts with #! and is followed by the interpreter path:

#!/usr/bin/perl

Sometimes we see the env program, which finds the the first perl in our path:

#!/usr/bin/env perl

env does not split args therefore we can’t add options:

#!/usr/bin/env perl -w

And, env is not always located in /usr/bin/env so it can guarantee some portability at machine/distribution level but not always between distributions.

Perl is nice

The perl is not like other interpreters—its nice, even with challenges. perl inspects the shebang to check if it’s really for it (and if not it hands our program over to another interpreter).

For instance the file i-am-python.pl contains a Python program, which is definitely not Perl:

#!/usr/bin/python
import os
import time

print("I'm a snake : " + os.environ["SHELL"] + " " + os.environ["_"])

# Keep it alive to have time to inspect with ps
while True:
    time.sleep(5)

Obviously we don’t care about the extension as it does not mean any kind of file association (although some systems let you associate it). So we have a .pl file and we execute it with perl but inside we have a python shebang and some python code. It’s clearly not a valid Perl file.

If you don’t believe me, check this with a quick syntax check perl -c i-am-python.pl that tells us it isn’t valid Perl:

$ perl -c i-am-python.pl
syntax error at i-am-python.pl line 3, near "import time"
i-am-python.pl had compilation errors.

When we execute this file with perl, surprisingly, everything goes fine. How did that happen? perl is smart enough to give the script to python!

$ perl i-am-python.pl
I'm a snake : /bin/bash /usr/bin/perl

And if we want to check which interpreter really runs this script, we can look in the process table:

$ ps aux | grep "i-am-pytho[n].pl"
tduponc+  5647  0.0  0.0  33208  7024 pts/0    S    13:04   0:00 /usr/bin/python i-am-python.pl

Note that i-am-pytho[n].pl with the brackets, which puts the n in a character class. That’s a nifty trick so grep finds the line with python but not the grep process itself because that pattern won’t match a literal [.

Don’t forget to kill the program since it’s sleeping forever!

Now, what if we want to test the converse and run Perl code with a python interpreter?

#!/usr/bin/perl

my $str = "I'm a jewel";
print "$str : $ENV{SHELL} $ENV{_}\n";

while (1) { sleep 5; }

This is a valid Perl file but the python interpreter does not hand over to perl and just returns a Python error:

$ python i-am-perl.py
  File "iamperl.py", line 3
    my $str = "I'm a jewel";
       ^
SyntaxError: invalid syntax

This is special to Python. Try it yourself with bash, Ruby, or something else.

I have something for you

Having the correct interpreter on the command line does not mean that the shebang is totally ignored. perl is once again super smart and behaves exactly as we can imagine (DWIM). For instance, what if we put a warning switch (-w) in the shebang, like in this file override-bang.pl:

#!/usr/bin/perl -w

$str = "will produce a warning";

Even though we don’t put the -w on the command line, we still get warnings:

$ perl override-bang.pl
Name "main::str" used only once: possible typo at override-bang.pl line 3.

Plenty is no plague

Now, what if we specify some switches on the command line and some others in the shebang? SPOILER: they are simply merged together.

When we run perl -c overridebang.pl to check a syntactically-valid file, we get the switches from the command line and the shebang line. We get a perl -cw execution:

Name "main::str" used only once: possible typo at override-bang.pl line 5.
override-bang.pl syntax OK

What if we have conflicting options like -w to enable warnings and -X to disable them? Here’s enable-warnings.pl:

#!/usr/bin/perl -w

$str = "will produce a warning";

When we run this on its own, we get a warning as expected:

$ perl enable-warnings.pl
Name "main::str" used only once: possible typo at warnings.pl line 3.

When we add -X on the command line, there is no output:

$ perl -X enable-warnings.pl

How about the other way around with -X on the shebang? Here’s disable-warnings.pl:

#!/usr/bin/perl -X

$str = "will produce a warning";

When we run this with -w, we still don’t get output:

$ perl -X enable-warnings.pl

The -X always turns off warnings.

The shebang (-X) is taken in priority versus the command line and no warning is reported. It’s the same if we execute the file with perl -W disable-warnings.pl.

We could imagine that’s a rule to resolve conflicts with “last seen” parameter but wait, it’s not that simple.

How about -X versus -W, which enables all warnings? Who wins then? It turns out that the last on defined wins. We can see that right on the command line:

$ perl -W -X -e '$str = "will produce a warning"'
$ perl -X -W -e '$str = "will produce a warning"'
Name "main::str" used only once: possible typo at -e line 1.

As an exercise for the reader, try the different combinations of taint checking options: -T and -U.

A magic incantation

Sometimes we see some odd lines at the beginning of Perl programs. What the hell is this black magic? This is actually very smart opening is “polyglot” and correct for both shells (with or without shebang support) and perl:

#!/usr/bin/perl
eval 'exec /usr/bin/perl -S $0 ${1+"$@"}'
    if $running_under_some_shell;

If we start the script with perl, the job is done and perl executes:

eval 'exec /usr/bin/perl -S $0 ${1+"$@"}'
if $running_under_some_shell;

That $running_under_some_shell has no value, so the code translate to a false conditional. This line is ignored and the rest of the file is interpreted normally.:

eval 'exec /usr/bin/perl -S $0 ${1+"$@"} if 0;'

What if we start the script with a shell that recognizes the shebang? The shell does the handover to perl, which then reads the first line (shebang then eval ...). The execution flow is then the same than above (magic incantation does nothing and file is interpreted). Nothing surprising there.

But what if we started the script with a shell that does not recognize the shebang so no handover occurs right away? This is actually where this magic is useful. The shell will ignore first line and will never reach third line. Why will it never reach third line? A newline terminates the shell command and exec will replace the current execution by perl. The rest of the script doesn’t matter after that exec. Our code changes from this:

#!/usr/bin/perl
eval 'exec /usr/bin/perl -S $0 ${1+"$@"}'
    if $running_under_some_shell;

to effectively this:

eval 'exec /usr/bin/perl -S $0 ${1+"$@"}'

Those $0 and $@ are shell words for the script name and arguments and the -S tells perl look for the value in $0 using PATH environment variable. (perldoc)

-x is fun

We’ve had fun with the perl interpreter and the shebang, but perl has a -x which is already fun by design. This option tells Perl that the program to execute is actually embedded in a larger chunk of unrelated text to ignore. Perhaps the Perl program is in the middle of an email message:

"I do not know if it is what you want, but it is what you get.
        -- Larry Wall"

#!/usr/bin/env perl

print "perl -x ignores everything before shebang\n";
print <DATA>;

__END__

"Fortunately, it is easier to keep an old interpreter around than an
old computer.
        -- Larry Wall"

Executing this as a program is a syntax error because the Larry Wall quote before the shebang is not valid Perl. When we execute this code with perl -x, everything before the shebang is ignored and it works:

$ perl -x email.txt
perl -x ignores everything before shebang

"Fortunately, it is easier to keep an old interpreter around than an
old computer.
        -- Larry Wall"

Out of curiosity, what if we tried to go one step further? How about multiple shebangs in a file, where one of them has a -x:

#!/usr/bin/perl -x
#!/usr/bin/perl

But it only produces an error:

Can't emulate -x on #! line.

There is however a trick to achieve this, by using shell eval. That perl -x is now executed in a shell process and not interpreted by perl binary like previously.:

#!/bin/sh
eval 'exec perl -x $0 ${1+"$@"}'
die "another day"; exit 1
#!perl
print "$]\n";

startperl

This article would not be complete without discussing a bit about the config variable $Config{startperl}. This variable comes from Config.pm that provides information about configuration environment (which you also see with perl -V):

$ perl -e 'use Config; print $Config{startperl}'
#!/usr/bin/perl

This is actually built during compilation from defaults or user/vendor provided configs. What if we want a different value? Simply specify the value of this during the ./Configure step, the configure option is -Dstartperl='...'. We then need to rebuild perl:

$ ./Configure -des -Dstartperl='#!/my/shebang'
$ make test install

Now our custom value is the default:

$ perl -e 'use Config; print $Config{startperl}'
#!/my/shebang

ExtUtils::MakeMaker and Module::Build seems also to use startperl among other methods to fix modules shebangs.

Take care to use an interpreter or a program that behaves like a perl interpreter! Some CPAN modules use startperl to write first line of generated perl tests. The /usr/bin/env limitation still apply here.

Resources

The #! magic, details about the shebang/hash-bang mechanism on various Unix flavours
Why it is better to usr /usr/bin/env interpreter instead of /path/to/interpreter
Could someone explain this shebang line which uses sh and then does exec perl?
A tiny portion comes from Sortie de Perl 5.30.0 (en français).

Stupid DATA Tricks

Mon, 05 Oct 2020 19:34:55 +0000

Image credit: Rob Nguyen on Flickr

I’ve previously written about Stupid Open Tricks, so now it’s time for some stupid DATA tricks. You probably already know that you can “embed” a file inside a Perl program then read it from the DATA filehandle. David Farrell wrote about this in Perl tokens you should know and he’s the one who reminded me about the curiousity that I’ll demonstrate here.

Anything after the __DATA__ line is not part of the program but is available to the program through the special DATA filehandle:

#!/usr/bin/perl
print "---Outputting DATA\n", <DATA>, "---Done\n";
__DATA__
Dog
Cat
Bird

The output shows each line after __DATA__:

---Outputting DATA
Dog
Cat
Bird
---Done

I typically go the other way by starting with a data file and adding a program to the top of it:

#!/usr/bin/perl
use v5.26;
use Text::CSV_XS;

my $csv = Text::CSV_XS->new;
while( my $row = $csv->getline(*DATA) ) {
	say join ':', $row->@[3,7];
    }
close $fh;

__DATA__
...many CSV lines...

This is the end, my friend, the END

You probably also know that you can use __END__ instead. I’m used to using that because it’s a holdover from Perl 4 and that’s where I first learned this:

#!/usr/bin/perl
print "---Outputting DATA\n", <DATA>, "---Done\n";
__END__
Dog
Cat
Bird

You get the same output:

---Outputting DATA
Dog
Cat
Bird
---Done

But now let’s get a little tricky. Define a package at the end of the program. This still uses __END__:

#!/usr/bin/perl
print "---Outputting DATA\n", <DATA>, "---Done\n";
package not::main;
__END__
Dog
Cat
Bird

Again, this outputs the same thing as before. Nothing surprising here, but the suspense must be building:

---Outputting DATA
Dog
Cat
Bird
---Done

Change that __END__ to __DATA__ and try again:

#!/usr/bin/perl
print "---Outputting DATA\n", <DATA>, "---Done\n";
package not::main;
__DATA__
Dog
Cat
Bird

Now you don’t see those lines:

---Outputting DATA
---Done

If you’ve read the documentation and cared about this sort of thing (or like me, forgotten it), you may have noticed that the DATA handle lives in the package that’s in scope at the end of the program:

Text after DATA may be read via the filehandle “PACKNAME::DATA”, where “PACKNAME” is the package that was current when the DATA token was encountered.

I can use the package specification to get the lines back:

#!/usr/bin/perl
print "---Outputting DATA\n", <not::main::DATA>, "---Done\n";
package not::main;
__DATA__
Dog
Cat
Bird

Now those lines are back:

---Outputting DATA
Dog
Cat
Bird
---Done

But what about the __END__? Well, that was a Perl 4 thing, before there were packages. Perl 5 added packages, then Perl 5.6 added the __DATA__ token. The __END__ kept doing what it was doing in the way it was doing it (package-less), and __DATA__ did something related by new:

For compatibility with older scripts written before DATA was introduced, END behaves like DATA in the top level script (but not in files loaded with “require” or “do”) and leaves the remaining contents of the file accessible via “main::DATA”.

Some other DATA tricks

There’s a few other interesting things you can do.

Program size

You can get the entire file size with the -s file test operator. The __DATA__ (or __END__) has to be there, but you don’t need any data after those tokens.

use v5.10;
my $size = -s DATA;
say "File size is $size";
__DATA__

The file size this reports includes everything in the file, not just the part before the end of processing.

use v5.10;
my $size = -s DATA;
say "File size is $size";

my $data = tell DATA;
say "Data starts at $data";
say "Data size is ", $size - $data
__END__
Dog
Cat
Bird

The program size includes the __END__ token and the newline after it. The rest belongs to the data:

File size is 164
Data starts at 151
Data size is 13

You can use DATA in other file things, including stat.

Read it twice

If you want to read the data twice, you can reset the file cursor. First, remember where DATA starts by calling tell before you read any lines. When you are ready to read it again, seek to that same position:

#!/usr/bin/perl
my $data_start = tell DATA;

print "---Outputting DATA\n", <DATA>, "---Done\n";

seek DATA, $data_start, 0;
print "---Outputting DATA\n", <DATA>, "---Done\n";
__END__
Dog
Cat
Bird

Using line numbers

#!/usr/bin/perl
my $data_start = tell DATA;

print "---Outputting DATA\n";
while( <DATA> ) {
	print "$. $_"
	}
print "---Done\n";

__END__
Dog
Cat
Bird

Now you see some line numbers, but those start counting from the first line under __DATA__:

---Outputting DATA
1 Dog
2 Cat
3 Bird
---Done

To get the real line numbers, you can figure out where the __END__ token is. This assumes that it’s not in the middle of documentation or in a string:

#!/usr/bin/perl
my $data_start = tell DATA;

my $end_line;
UNITCHECK {
	open my $fh, '<', $0;
	while( <$fh> ) { last if /\A__END__$/ }
	$end_line = $.
	}

print "---Outputting DATA\n";
while( <DATA> ) {
	$n = $end_line + $.;
	print "$n $_"
	}
print "---Done\n";

__END__
Dog
Cat
Bird

Now you see the offsets in the whole file and not the count after the __END__:

---Outputting DATA
19 Dog
20 Cat
21 Bird
---Done

There are some more vigorous methods in Can a Perl program know the line number where DATA begins? .

Multiple embedded files

This isn’t a DATA thing, but you can make several embedded files with Inline::Files:

#!/usr/bin/perl
use Inline::Files;

print "---Outputting dogs\n", <DOGS>, "---Done\n";
print "---Outputting cats\n", <CATS>, "---Done\n";
print "---Outputting birds\n", <BIRDS>, "---Done\n";
__DOGS__
Rin Tin Tin
Lassie
Ol' Yellar
__CATS__
Grumpy Cat
Garfield
Maru
Mr Bigglesworth
__BIRDS__
Woody Woodpecker
Roadrunner
Zazu
Sam the Eagle

Each of those get their own filehandles:

---Outputting dogs
Rin Tin Tin
Lassie
Ol' Yellar
---Done
---Outputting cats
Grumpy Cat
Garfield
Maru
Mr Bigglesworth
---Done
---Outputting birds
Woody Woodpecker
Roadrunner
Zazu
Sam the Eagle
---Done

Inline::Files has a problem because it overrides open. You have to use CORE::open to get to the real one:

use Inline::Files;

print "---Outputting dogs\n", <DOGS>, "---Done\n";
print "---Outputting cats\n", <CATS>, "---Done\n";
print "---Outputting birds\n", <BIRDS>, "---Done\n";

CORE::open my $fh, '<:utf8', '/etc/hosts' or die $!;
print "---Outputting hosts\n", <$fh>, "---Done\n";

The Perl Ambassador: Damian Conway

Thu, 01 Oct 2020 00:00:00 +0000

This month I interview Damian Conway, one of the Guardians of Perl. Damian is computer scientist and excellent communicator—his presentations and courses are widely popular around the world. He was the Adjunct Associate Professor in the Faculty of Information Technology at Melbourne’s Monash University between 2001 and 2010.

It was an honour to interview my idol. I enjoyed talking to him and I am sure you would have many “aha” moments. For example, Raku’s built-in grammar construct is inspired by the work of Damian’s Parse::RecDescent.

I also found out that Damian was the winner of Larry Wall Award for Practical Utility three years in a row from 1998 to 2000 for contributions to CPAN.

He also wrote some of the most popular books on Perl:

If you’d like me to interview you, or know someone you’d like me to interview, let me know. Take the same set of questions and send me your answers!

When and how did you get started with Perl?

In the mid-to-late 1990s, I was an academic at Australia’s largest university. At the time I was working on an unusually diverse range of research topics: educational technology, programming language design, software engineering, documentation systems, human-computer interaction, natural language generation, emergent systems, image morphing, human-computer interaction, API design, geometric modelling, the psychophysics of perception, nanoscale physical simulation, and parsing techniques. I liked to think I was a polymath, but the correct word is probably “dilettante”.

Anyway, I was implementing most of my software in C++, and I desperately needed some other language (any other language!) that would make text manipulation and generation, interface design, and programming language research a great deal easier. In particular, I was trying to solve problems in natural-language text generation, and so I searched the infant internet for a programming platform that would make that task, if not easy, then at least not painful. During that search I found Perl.

Within six months I had submitted two papers (on automating English inflexion, and on declarative command line interfaces) to the 2nd Perl Conference. Both were accepted and so I received funding from my institution to attend the conference. I don’t think I had been at the conference for more than half a day before I realized that I had finally found my tribe. And the rest is history.

What made you write Perl Best Practices?

Mostly it was two of the three Perl virtues: Impatience and Hubris. In the early 2000s I was becoming extremely impatient with the prevailing view outside the Perl community that Perl was a “write-only” language that produced code which was impossible to read, understand or maintain. And I was equally impatient with the prevailing view inside the Perl community that “More Than One Way To Do It” was a licence (or possibly a challenge) to do it in the most terse, tricky, and obscure ways possible.

Which led to hubris: the notion that I could find and teach a set of better habits for Perl coding; habits that would automatically produce code that was cleaner, more readable, more understandable, more robust, and more maintainable that the Perl code I was then seeing (well, to be honest, that I was then seeing mostly in my own terse, tricky, obscure module implementations).

So I sat down and reviewed every line of Perl code I had ever written, and a great many lines of Perl code that others had written, trying to find patterns of coding that consistently led either to excellent or to terrible code. Within six months I had over 250 such observations, and started teaching them in a two-day class. Soon after that, O’Reilly got in touch and suggested they might make a good book.

Do you follow your own suggestions in the book?

Yes. I feel it would be highly hypocritical of me not to.

However, I’ll admit that, at first, eating my own dog food was pure agony. A solid majority of my 250+ pieces of advice advised behaviours that I was not following myself at the time. I had to change just about everything from my curly-bracketing style (I was a confirmed BSD layout partisan, but PBP advocates K&R style), to my identifier-naming conventions (I originally didn’t have any at all, and was highly addicted to single-letter variables), to my commenting style (“Comments? My code is self-documenting!”), to the way I wrote OO Perl (usually without a framework, and always without proper encapsulation).

It took me months to replace all my bad habits with better ones, and even longer to download those new habits into my fingertips, so that they “just happened” whenever I was typing in code. But it’s been so very worthwhile. I can clearly see the difference between my pre-PBP code and the code I wrote after that time. I actually enjoy maintaining and improving the more recent code, whereas I find myself continually putting off any need to delve back into my older codebases.

In several cases (for example, Lingua::EN::Inflect or IO::Prompt) it was easier to completely rewrite the module (creating Lingua::EN::Inflexion and IO::Prompter respectively) rather than deal with that dreadful pre-PBP code.

Of course, there’s nothing intrinsically magical about the particular suggestions in the book. Indeed, in my current version of the class, somewhere around 20% of my advice has actually changed from what I wrote in PBP. The magic is in adopting and practising any set of consistent, well-thought-out, and productive coding habits.

What do you think about “Perl 7” currently being discussed widely?

I think it’s great to see Perl moving out from under the lingering ghost of “Perl 6”. And to see such a strong statement of positive forward motion, hopefully without too much of the attendant disruption of breaking vast swathes of existing code.

And, more importantly, I think it’s vital that the widely accepted boilerplate components of a modern Perl program (use strict, use warnings, declarative subroutine parameters, postfix dereferencing) should be turned on by default under new versions of Perl. And, of course, that some of the especially problematic vestigial components (indirect-object method calls, implicit hash-key concatenation, bareword filehandles) should be deprecated with extreme prejudice.

Of course, a plan this bold and this unanticipated will inevitably create anxiety and raise dissent. And not all those fears and disagreements will be misplaced. Even so, the very best thing about Perl 7 (whatever it ultimately proves to be) is that even just the idea of making a major version bump to usher in such fundamental changes is generating a huge amount of productive discussion and debate with the Perl community, and injecting a vast quantity of new energy, which must surely eventually lead to a better outcome, a better path forward for Perl.

Do you follow the development of “Cor”?

Yes, I’ve been following the Cor project with great interest for the past six months or so. I’ve occasionally also been consulting on the design of this proposed new mechanism, discussing particular issues and offering suggestions to Ovid on several aspects of the design. (Editor: See our earlier interview with Ovid)

Needless to say, I’m immensely excited by the prospect of having a genuinely declarative interface for class definitions added right into the core of the language! Even if that happy day is likely to be a few years away yet.

You have loads of handy modules published on CPAN. What are your top 5 contributions and why?

In no particular order:

Lingua::EN::Inflexion

This is the successor to my first ever public Perl module (Lingua::EN::Inflect), which was one of the two modules I presented at my first ever Perl conference, and which therefore in a sense paved the way for all the others. Both the original and the successor modules attempt to solve the very challenging problem of producing and recognizing correct word inflections in English (“cat”→”cats”, “box”→”boxes”, “ox”→”oxen”, “fish”→”fish”, “goose”→”geese”, “maximum”→”maxima”, “penny”→”pennies”, etc. etc.)

The reason I’m nominating the successor module instead of the original as one of my top five is that Lingua::EN::Inflexion is vastly more powerful, more reliable, and more maintainable, and reflects so much of what I have learned about the design and implementation of good APIs over the past two decades.

Parse::RecDescent

This was another early module, which provided what I like to think was the first truly Perlish grammatical parsing engine for the language. Prior to its release there had been ports of lex/yacc for Perl, but Parse::RecDescent allowed you to define grammars directly in your program, directly in Perl. It would also execute those grammars immediately, without a separate precompilation phase. The module has been very widely used over the past two decades in a huge number of commercial and academic projects. Finally, many of the ideas I pioneered in Parse::RecDescent eventually made their way into Raku’s built-in grammar construct, a contribution of which I’m immensely proud.

Quantum::Superpositions

In a similar vein, I have a huge fondness for my Quantum::Superpositions module. Not only because my presentation on that module has always been so hugely enjoyable to give, but also because that module led directly to the fundamental Raku concept of “junctions”, which I feel might be my only meaningful contribution to the field of computer science.

Regexp::Debugger

I use a great many regexes in my Perl solutions. When they go right, regexes are such powerful and efficient tools for solving complex data-processing problems, but when they go wrong, they’re an swirling abyss of confusion and frustration. That’s why I feel that Regexp::Debugger is probably the single most useful module I’ve ever written. It changes the task of debugging complex regexes from a multi-level, multi-day, multi-pain-killer nightmare into a simple, quick, “d’oh”-moment of sudden revelation. This module has saved me (and many other Perl users, I hope) literally thousands of hours of misery, simply by making our regex mistakes directly visible from right within the code itself.

PPR

Talking of complex regexes, I love the PPR module because it consists almost entirely of a single regex; a single 70000-character regex that can parse the vast majority of valid Perl documents. It’s such a mind-bogglingly simple and obvious idea; and yet it was so damn difficult to actually achieve. But worth every long hour of struggle. Having the ability to write a regex that matches a complete block of code, or a variable declaration, or a ‘for’ loop, or a ‘use’ statement, or any other Perl construct, opens up an entirely new universe of possibilities for validating, modifying, and refactoring Perl code. PPR is the basis for so many of the other CPAN modules of which I’m most proud, especially Keyword::Declare (which allows you to safely extend the Perl syntax using Perl itself), Dios (which brings most of Raku’s vastly superior OO model and syntax to Perl), and Code::ART (which provides Perl-refactoring tools for the Vim editor).

I think my most memorable single experience in Europe was lecturing on Raku (or, rather, on Perl 6, as it was back then) at CERN in Geneva in 2015.

It was a pure joy to be in a room with so many brilliant people, talking with them about a project I have been so deeply committed to over the past two decades. And to watch that room full of brilliant people start to overcome their initial reluctance and begin to appreciate just how powerful this new language can be…that was a great moment.

Of course, it did require some chicanery to encourage them to put aside their scepticism and turn up in the first place. Specifically, I rearranged my standard talk so that I didn’t actually use the word “Perl” at all for nearly ninety minutes. In fact, I avoided naming the language in any way until the second-to-last slide.

And, indeed, it was that particular experience–seeing how easy it was to sidestep the widespread general prejudice against anything related to “Perl”–that made me so very keen to support the name change from “Perl 6” to “Raku” last year.

That visit to CERN was special to me for another reason too. Before my presentation, I had the privilege of being taken on a private tour of the facility. And though I didn’t manage to get down into the LHR itself (it was closed at the time for a refit), I did have my mind utterly blown by being allowed to wander around a building that seemed to be straight out of science fiction: the Antimatter Factory.

You briefly blogged about the weekly challenge, how did you find about it?

I learnt about the Weekly Challenge from the many blog posts from participants, which suddenly started popping up on my blogs.perl.org and reddit.com/r/perl+rakulang blog feeds. They seemed to be having so much fun that I couldn’t resist trying it myself.

And what was your experience with it?

I loved taking part in it. Not just for the individual challenges it offered me, but also for the opportunities it gave me to showcase some of the power, expressiveness, and convenience of solving those puzzles in Raku. And in doing so to reach out beyond the Perl and Raku communities to briefly capture the attention of the wider developer world.

It’s well known how much I adore teaching, and the Weekly Challenge gave me yet another opportunity and outlet to indulge that passion.

Where do you get the ideas for your next CPAN contributions?

I think most of my ideas come directly from my extremely low threshold of annoyance. Whenever I am in the middle of trying to solve a particular problem in Perl, and I can’t find a sufficiently simple way of doing so, I frequently get quite unreasonably frustrated. And my immediate response is to ask myself: “If I could make this work any way I wanted, how would I make it work?”

And then I remember that, being moderately competent in Perl, I can make it work in any way I want. So I do.

Mostly the actual ideas and solutions I implement come from a process I call “Design by Coding”. That is, when something is insufficiently easy, I just go ahead solve the problem the way I wish I could solve it, by writing the code that I wish I could write.

Of course, that doesn’t actually solve the problem, because this “gedankencode” won’t actually run. But then I merely need to spend a ridiculous and completely unjustifiable amount of time and energy making the hypothetical code work in real life. Typically, far more time and energy than it would have taken to just do it the hard way.

But the benefit is that, having expended all those hours on creating a better solution, thereafter I never have to do it the hard way again. And because I always put those solutions on CPAN, neither does anyone else.

At a deeper level, I think I’ve been able to come up with so many unusual ideas in my career because I spend a great deal of time “filling the well”. My wife is an artist, and she introduced me to this concept: that in order to be creative, you have to constantly refill your mind with a large number of novel and random facts, observations, and ideas. Creativity is then the process of finding new and unexpected connections between those ideas. The more ideas you have, the more links you can easily find, so the more creative you can be.

And that’s what I try to do. I try to read as diversely as I can, to give my creative processes the greatest possible range of building blocks and the greatest number of plausible (and implausible!) connections.

What would you suggest to someone starting Perl?

Three things:

1. Write bad code.
2. Study better code.
3. Read great books.

I think the fundamental mistake most beginners make, however, is that they try to do those three things in the wrong order: books first, other people’s code next, writing their own code last.

I don’t think there is any substitute for writing your own code from the very beginning. You need to grapple immediately with the challenges and puzzles of a new language. It’s the only way to actually improve as a programmer, especially in a language as eclectic and idiomatic as Perl.

Only after you have struggled to write Perl yourself, you can start to appreciate the better code that others may have written. And once you can begin to recognize good Perl code, you can start to read productively about Perl, to learn why that code is good.

Perl has so many great textbooks, but very few of them are completely sufficient if you’re not already at least a little familiar with the language itself. Without that prior struggle, I don’t think you can genuinely appreciate the “aha!” moments, the sudden opportunities for deep understanding, that a truly excellent textbook will offer you.

One other point I would make is that Perl almost suffers from a surfeit of excellent introductory textbooks…which can make it difficult for any particular beginner to find the one best suited to their own background, their personal learning style, and their individual goals.

Learning Perl might be perfect for one person, but someone else might do much better with Modern Perl. Some folks will find that a pragmatic use-case approach like Impatient Perl best meets their needs, whereas others would be far better served by a text that’s at almost the opposite extreme, like the CS-based theoretical framework of Elements of Programming with Perl.

The point is: if you’re looking for a book that will help you understand Perl, you should first try out at least a few of the many excellent alternatives available so you can find the one book that will help you understand Perl.

I noticed you are associated with Raku these days. Do you still use Perl?

Almost every day. I love Raku dearly, and I’m starting to use it more and more often in my daily coding, but when I need to get something critical implemented ASAP and then have it run as fast as possible on as wide a range of platforms as possible, Perl is still the tool I reach for almost every time.

For larger projects, Raku has some unbeatable advantages and in the next few years I expect its performance to become far more competitive, but for quick-and-dirty get-it-done-now tasks, Perl is still my preferred chainsaw.

And I suspect it always will be.

Searching Internet RFCs

Thu, 24 Sep 2020 07:28:49 +0000

During the quarantine I was able to find the good side of the home confination: I hadn’t enough time to read a book due to school’s tests, but for my luck, I had enough time for reading one or two Request For Comments (RFC) documents.

Since my first days studying computer security, the concept of “protocol” fascinated me. Maybe for their enormous diffusion in almost every computer system, our daily lives heavily depends from these processes. As I say “trust on machines but don’t trust humans”. The RFC approach reminds the open source philosophy, which has the same objective (give everyone the opportunity to learn new things through sharing) and the same propagation channel: the internet.

I find it too hard to search for these documents on the IETF website, so I made a fast and efficient script that permits me to download RFCs through a keyword and lets me decide which ones to read and which ones to ignore.

What are RFCs?

“Request For Comments” are one of the best online references for learning how networking really works at the protocol level. These are simply text documents that define every single standard that has been approved by the Internet Engineering Task Force. These documents are a good entry point for developing the study of protocol analysis: an emerging field that uses formal verification to find exploits.

Every RFC has a title and its associated number, for example the RFC 3607 is the “Chinese Lottery Cryptanalysis Revisited: The Internet as a Codebreaking Tool”.

My script uses Net::RFC::Search for the search, and Term::ANSIColor for the display—there will be a lot of text in the terminal and different coulrs can help!.

#!/usr/bin/perl
# Made by Edoardo Mantovani in 2020

use strict;
use warnings;

use File::Path qw(make_path);
use File::Spec::Functions;
use LWP::Protocol::https; # Net::RFC::Search needs this
use Net::RFC::Search;
use Term::ANSIColor;

my $folder = shift or die "./RFC <Folder> <Query>\n";
my $query  = shift or die "./RFC <Folder> <Query>\n";

make_path $folder;

my $rfc_interface = Net::RFC::Search->new();
my @rfc_nums = $rfc_interface->search_by_header( $query );

print "RFCs [@rfc_nums]\n";

foreach my $rfc_num ( @rfc_nums ) {
  my $local_file = catfile( $folder, $rfc_num ); # i.e final folder = /tmp/1110
  if ( $rfc_interface->get_by_index( $rfc_num, $local_file ) ) {
    print colored( "Downloaded: $rfc_num\n", "green" );
    view_rfc( $local_file );
  }
  else {
    print "Error with RFC $rfc_num\n";
  }
}

sub view_rfc {
  my( $file ) = @_;

  open my $fh, '<', $file or do {
    warn "Could not open file [$file]: $!\n";
    return;
  };

  while( <$fh> ) {
  	print;
  	next unless $. == 25;
    print colored("Do you want to save this RFC? [Y/N] ", "red");
    my $input = <STDIN>;
    chomp( $input );
	unlink $file if uc($input) eq "N";
    last;
  }
}

Running the program requires a directory name and a query. I create the directory if it doesn’t already exist:

$ ./rfc_search rfcs cookie

I display the first 25 lines of each matching RFC and prompt to save it.

How it works

The search_by_header returns the RFC numbers for each of the documents that has the query word:

my @rfc_nums = $rfc_interface->search_by_header( $query );

I then fetch each document by specifying its number and local filename. I do this even if I won’t keep it because I don’t know what’s in it yet:

$rfc_interface->get_by_index( $rfc_num, $local_file );

For the last part, I look at the first 25 lines of the file. I have a prompt that asks me if I want to save it. If I enter “N”, I delete that file. I color the prompt so I can see it against the RFC text:

sub view_rfc {
  my( $file ) = @_;

  open my $fh, '<', $file or do {
    warn "Could not open file [$file]: $!\n";
    return;
  };

  while( <$fh> ) {
  	print;
  	next unless $. == 25;
    print colored("Do you want to save this RFC? [Y/N] ", "red");
    my $input = <STDIN>;
    chomp( $input );
    rmtree(@_) if uc($input) eq "N";
    last;
  }
}

Other uses and suggestions

I made this simple script mainly for save time and have a better interface for interacting with the RFC site, I want to recommend to see the TODO part in the Net::RFC::Search page. For a better script, it would be cool to implement curl and lynx to retrieve cancelled RFC’s. Of course the script can be implemented with custom functions but this is script is only for educational purposes.

resources - Request for Comments: User Stories and Scenarios - Wikipedia’s RFC entry

The Perl Ambassador: Curtis 'Ovid' Poe

Sat, 08 Aug 2020 04:40:00 +0000

This month’s interview is Curtis ‘Ovid’ Poe, one of the most-respected and well-known leaders in the Perl community.

Curtis has been building software for decades. He specializes in building database-driven websites through his global development and consulting firm, All Around The World. He’s the main developer behind Tau Station, a text-based Massive Multiplayer Online Browser Game (MMOBG) set in a vibrant, far-future universe.

He’s the author of Beginning Perl and Perl Hacks. You can out more about his activities at https://ovid.github.io/.

He joined The Perl Foundation boards of directors in 2009.

If you’d like me to interview you, or know someone you’d like me to interview, let me know. Take the same set of questions and send me your answers!

How did you first start using Perl?

I first started using Perl about 20 years ago, when I was doing mainframe programming. I was trying to fix a problem with a COBOL program that was converting a CSV file from an NT system to the fixed-width format that COBOL prefers. COBOL has many weaknesses and working with text is one of them. The code was 150 lines long, but that’s because the author didn’t understand how the COBOL’s unstring function worked. I got it down to 80 lines of COBOL. Out of curiosity, I tried it in Perl and got it down to 10 lines of code. Everything I touched in Perl was shorter and easier to read, so I jumped ship.

What do you think about Perl 7? Do you see the Perl is on the path of recovery?

I am 100% on board with the project and yes, it’s the path for Perl’s recovery. I’ve seen widespread support for the change, which was heartening, and with the amount of press, even TIOBE moved Perl from 19 to 14th place, though I suspect it will drop back after the press dies down.

However, there’s a difference between having a goal and having a plan. There’s widespread agreement on the goal, but there’s less agreement about the plan. That’s great so long as people can use this to find the best path. It’s less great if it devolves into acrimony. Fortunately, Sawyer’s been great at projecting a positive message.

So long as we manage to protect businesses currently using Perl (and that means convincing Linux distros that we’re not going to break them), having a plan to better support active and new developers is awesome. And the version number change is a key first step.

What inspired you to start the project “Cor”? When are you planning to release it?

First, for those who are not familiar with Cor, you can read about it at https://github.com/Ovid/Cor/wiki. In short, Cor is a plan to add modern OO to the Perl core. But the motto is “‘Good enough’ is not good enough.” We have to stop settling for what we can have and start dreaming about what we want. For a trivial example, here’s a naïve LRU cache implementation in Cor:

class Cache::LRU {
    use Hash::Ordered;

    has $max_size :new(optional) :reader :isa(PositiveInt) = 20;
    has $created  :reader = time;
    has $cache    :handles(get)  :builder;
    method _build_cache () { Hash::Ordered->new }

    method set ( $key, $value ) {
        if ( $cache->exists($key) ) {
            $cache->delete($key);
        }
        elsif ( $cache->keys > $max_size ) {
            $cache->shift;
        }
        $cache->set( $key, $value );  # new values in front
    }
}

As for my motivation, like many developers, I was waiting for Stevan Little to finish his work to get a better OO system for the Perl core. But he was working mostly alone, for a long time, and that’s hard to do. So when I decided I wanted something, I was able to take a look at his work and realize it was solid. But I needed a better syntax.

I tried to refine some of the syntax from Moo/se, but honestly, Moo/se has some serious limitations. Some are design decisions which can be easily corrected, but some are due to limitations in the Perl language itself. Once I had Sawyer’s backing, I realized that I didn’t just have to steal syntax, I could invent syntax, though I have done so very cautiously. It’s important that Cor still be Perl, but just a tiny sprinkling of syntactic sugar in the right spots makes a world of difference. So far it looks promising.

And Sawyer said he hopes a v1 will be available under a feature guard in 7.2 or 7.4. With Perl 8, the feature guard would be removed.

How does your company “All around the world” help people with Perl?

The consulting world is a mess. Anyone can call themselves a consultant and presto, they’re a consultant. This means that for the vast majority of consulting firms out there, quality is very hit-or-miss. So most companies hiring consultants are taking a huge gamble. But banks, insurance companies, and other “enterprise” companies prefer to stick with high-end consulting firms. From what I’ve seen, their code is more likely to do what you want it to do, but that’s only if you can afford them.

So we decided to try a different approach. We can give our customers the reliability they want but at a more reasonable price if we do two things. First, we only hire the handful of senior developers who can pass both our technical test and a structured interview. Second, we limit the number of projects we take so we can dedicate serious attention to each customer. I’ll put our team’s quality up against the top-tier consulting firms any day of the week. But we’re going to cost a much less than they do and we’ll deliver faster, too.

To give a concrete example, we had a client who had two weeks to improve their performance by an order of magnitude. They had worked with us before, so they turned to us. Here’s the case study of that project. It’s a fun read and gives you a lot of insight into how top-level developers really work.

I should also note that while we have a deep specialization in Perl, we have also done work in quite a few other languages and technologies, such as Golang, C++, Lua, Node, Angular, and so on.

How is “Tau Station” going and how much Perl helped in building the game?

For those not familiar with it, Tau Station is a free to play narrative MMORPG with the backend written entirely in Perl. It’s a beautiful sci-fi universe (unlike anything you’ve ever played before) and has all the stars within 20 light years of Earth. We’re around half a million lines of code and we’re still in open alpha. We’re currently in the “final stretch” of building what we feel we need, so we anticipate the launch by the end of this year. Perl’s tremendous flexibility has made it very easy to build out many of the features that we’ve needed. For example, we have a declarative system for building out many of the behaviors. Here’s how you refuel a spaceship:

        Steps(
            Area(      $character => is_in            => 'docks' ),
            Ship(      $ship      => is_owned_by      => $character ),
            Ship(      $ship      => is_docked_on     => $character->station ),
            Character( $character => not_onboard_ship => $ship ),
            Ship(      $ship      => 'needs_refueling' ),
            Money(     $character => pay              => $ship->get_refuel_price ),
            Ship(      $ship      => 'refuel' ),
            Character(
                $character => set_cooldown => {
                    cooldown_type  => 'ship_refuel',
                    period         => $ship->get_refuel_time,
                },
            ),
        )

People are sometimes surprised to learn that this is Perl code because it’s so easy to read, but they’d be even more surprised to learn that much of this would be harder to write in early-binding languages such as Java.

And by creating standard components like that, the developer who creates a new kind of behavior for the game often doesn’t need to worry about database transactions, exceptions, or messages to the character. Instead, they can quickly assemble these “steps” in the correct order and you have new gameplay. If we ever had the time, we’d love to release the above framework as open source, but that would take time and we need to keep our clients happy, first.

Which Perl modules are you constantly using? How do they make your life easier?

Test::Class::Moose is a go to module for me. Most Perl developers learn how to test modules, not applications. With Test::Class::Moose, large test suites become easier to build and manage and, when it’s written correctly, many test suites can be an order of magnitude faster.

I also have a module I write for personal code called Less::Boilerplate. It’s not on the CPAN because it’s too fine-tuned for my personal needs, but naturally it enables strict, warnings, signatures, autodie, and other features without having to type everything out by hand. And it pleases me to have the double meaning of use Less::Boilerplate at the top of my code. Yet it’s part of the issue that Sawyer’s pointed out with Perl. New Perl developers don’t know the strange incantations experienced Perl developers put at the top of their code to get Perl to be reasonable. That hurts the language because they get a poor “out of the box” experience.

Which Perl feature do you overuse?

It used to be the punctuation variables that I would sprinkle around my code like magic pixie dust. Things like local $" = ', ' were natural to me. But I’ve stopped doing that because honestly, it’s not readable. I do a lot of client work so I take care to ensure that my code is (as much as I can), easy to read and maintain. I’ve even rewritten some of my code to “dumb it down” because I want to ensure that it’s maintainable.

Which Perl feature do you wish you could use more?

Given that clients call me in to help build new systems or fix existing ones, I pretty much get to pick and choose what features I want to use, so I’m fortunate in that regard. Thus, there’s not much I don’t get to use if I think it’s useful. However, a feature I wish I could use more is a feature that doesn’t exist: more standardized introspection tools, similar to a MOP. Mucking about in the symbol table for the things I need, or pulling in external libraries to find out where my code is located in the filesystem is frustrating. I often write code that magically “works” without having to be loaded (similar to plugins). But without standardized, cross-platform tools for finding the code, loading it dynamically, converting between package and filenames automatically, I find that I’m often rewriting this code again, for a different client, based on their operating system, file system layout, and so on.

What one thing you’d like to change about Perl?

How variables behave. There are a few things in that, but mainly, I wish Perl had invariant sigils like Raku and that arrays and hashes wouldn’t flatten unless requested. But I’m not going to get that, so let’s pretend I didn’t ask :)

(Hmm, working threads might be interesting, too)

What's new on CPAN - June 2020

Wed, 29 Jul 2020 01:10:28 +0000

Welcome to “What’s new on CPAN”, a curated look at last month’s new CPAN uploads for your reading and programming pleasure. Enjoy!

APIs & Apps

App::Timestamper::WithElapsed for every line of STDIN displays a timestamp and the elapsed seconds since the last line
Get a command line client for cPanel UAPI and API 2 using App::cpanel
Introspect C/C++ code with CastXML Clang::CastXML
OpenTelemetry supports application process monitoring as defined by opentelemetry.io
OpenTracing::Interface defines an API for opentracing (precursor to opentelemetry)
Send SMS via VoIP.ms with SMS::Send::VoIP::MS

Config & Devops

App::PP::Autolink can create standalone Perl executables, finding dynamic libs automatically
Get a simple interface for creating and verifying One Time Passwords as used by authenticator apps with Authen::TOTP
Guacamole is a parser toolkit for Standard Perl - also see Sawyer X’s recent talk
Neo4j::Client helps configure and build the C based neo4j-client library
Create and verify password hashes for OpenSMTPD using OpenSMTPD::Password

Data

Quickly extract raw values from Excel XLSX spreadsheets with Excel::ValueReader::XLSX
SkewHeap::PP is a fast and flexible heap structure
Text::Mrkdwn::Escape can escape text for inclusion in the markdown variant used by Slack
Perform HTTP Encrypted Content Encoding (AES 128-bit Galois/Counter Mode) with Crypt::RFC8188

Development & Version Control

Analyze/Rename/Track Perl source code, includes a vim plugin: Code::ART
Future::Buffer implements a Futures-based string buffer
Mojolicious::Command::Author::generate::cpanfile creates a cpanfile by scanning your source code for dependencies
Sys::Pipe provides access to the non-blocking pipe2() system call
Get fast and minimal code coverage stats using Test2::Plugin::Cover

Hardware

Get a chip driver for Noritake GU-D display modules using Device::Chip::NoritakeGU_D
PINE64::MCP300x provides an interface to the MCP300x family of 10-bit analog to digital converters
PINE64::MCP3208 provides an interface to the MCP3208 12-bit SPI analog to digital converters

Language & International

Getopt::EX::i18n sets the environment locale via a command line option
Translate::Fluent is a Perl implementation of the Mozilla localization project to create more natural-sounding translations

A tour with Net::FTP

Mon, 13 Jul 2020 11:48:52 +0000

When we want to have a way to exchange files between machines, we often think about rsync, scp, git or even something slow and complex (looking at you Artifactory and S3), but the answer is often right in front of your eyes: FTP!

The “File Transfer Protocol” provides a very simple and convenient way to share files. It’s battle-tested, requires almost no maintenance, and has a simple anonymous access mechanism. It can be integrated with several standard auth methods and even some virtual ones, none of which I show here.

In this article, I’ll install a local FTP server and create a simple FTP client in Perl.

A bit of context

At $work I have to carry on an army of developers that create customized build pipelines from handcrafted local configuration files.

This file is not hosted “by design” like you would have with Travis CI or a GitHub Action, but it is used to feed a “heavy client” that parses, resolves templates, and creates a workspace in some centralized automations servers through HTTP API calls.

It was a lot of support to help developers to create this file according to the spec (yet another file format), and we were blind when we wanted to help them with failing workspace creation/build (no way to retrieve configuration from workspace).

I got the idea to backup and centralize automatically the configuration file during the creation of the build pipeline workspace. It was intended to help both developers (configuration “samples”) and support team (see history, versioned then we can check diffs, file to replay). The constraints were to be able to exchange file from various places with variable users. The FTP protocol is a perfect fit for that.

I added also a cronjob to autocommit and push to a git repository and we had magically a website listing versioned configurations files.

In addition, FTP proved later to also require zero support. I mean really zero maintenance!

Download and install ftpd

I decided to use pure-ftpd but there are some other good alternatives if you want.

First I download the tarball, untar it, and change into its directory:

$ wget https://download.pureftpd.org/pub/pure-ftpd/releases/pure-ftpd-1.0.49.tar.gz
$ tar xvzf pure-ftpd-1.0.49.tar.gz
$ cd pure-ftpd-1.0.49/

I configure ftpd so I can execute it as casual (non-root) user using a non-restricted port, and I’ll set the destination directory to my $HOME/ftpd :

$ ./configure --prefix=$HOME/ftpd --with-nonroot && make && make install

I create two directories. The ftp is what I’ll publish and run is where I’ll put the pidfile.

$ cd $HOME/ftpd
$ mkdir ftp
$ mkdir run

Now we can start the ftp server. I need to give some custom configurations:

FTP_ANON_DIR is the directory I want to publish
-e allows anonymous access
-M allows anonymous users to create directories

-g specifies the directory for the pidfile:

$ FTP_ANON_DIR=`pwd`/ftp ; ./sbin/pure-ftpd -e -M -g run &

At this point I should have a running ftp server. Let’s check!

Test with ftp

First, I test with the preinstalled ftp client. If everything is fine I see the typical FTP exchange:

$ ftp localhost 2121
Connected to localhost.
220---------- Welcome to Pure-FTPd ----------
220-You are user number 1 of 50 allowed.
220-Local time is now 11:56. Server port: 2121.
220-Only anonymous FTP is allowed here
220 You will be disconnected after 15 minutes of inactivity.
Name (localhost:tib):
230 Anonymous user logged in
Remote system type is UNIX.
Using binary mode to transfer files.
ftp>

If I get ftp: connect: Connection refused it’s probably one of:

ftpd is not running (check with ps aux | grep "ftp[d]")
I’m using the wrong port

If I get 421 Can't change directory to /home/tib/ftpd/ftp/ [/] it’s probably because I haven’t created the directory I specified in FTP_ANON_DIR.

Simple client in Perl

Ok that’s cool, but I only played with ftp server and preinstalled ftp client until now. What about writing some Perl now?

Net::FTP is a superb CPAN module dedicated to FTP protocol and I’ll use that.

Simple listing

First, a very simple listing script ls.pl. This program connects to the server, asks for a list of files, and outputs each one. It’s clear that’s super easy and straightforward to play with FTP in Perl!

#!/usr/bin/env perl

use warnings;
use strict;

use Net::FTP;
my $HOST = "localhost";
my $PORT = 2121;


my $ftp = Net::FTP->new($HOST, Port => $PORT, Debug => 0)
	or die "Cannot connect to $HOST: $@";
$ftp->login() or die "Cannot login ", $ftp->message;
foreach my $f ($ftp->ls()) { print "$f\n"; }
$ftp->quit;

Upload

What next? Maybe upload something? Again, it’s super simple. Instead of listing files, I’m putting them:

#!/usr/bin/env perl

use warnings;
use strict;

use Net::FTP;
my $HOST = "localhost";
my $PORT = 2121;

my $ftp = Net::FTP->new($HOST, Port => $PORT, Debug => 0)
	or die "Cannot connect to $HOST: $@";
$ftp->login() or die "Cannot login ", $ftp->message;
foreach my $file(@ARGV) {
    $ftp->put("$file", "$file")
    	or die "Cannot put $file", $ftp->message;
}
$ftp->quit;

I run this and supply the files I want to upload:

$ perl upload.pl file1.txt file2.txt`.

Put things together

I propose a more complete client with some command-line parsing and more actions. In addition to the previous code for listing and uploading, here I added a way to view a file. Getopt::Long to handle command line parameters.

#!/usr/bin/env perl

use warnings;
use strict;

use Getopt::Long;
use File::Slurp;

use Net::FTP;
my $HOST = "localhost";
my $PORT = 2121;

my %options = ();

GetOptions(
	"action|c=s" => \$options{'action'},
	"file|f=s"   => \$options{'file'},
	"help|h"     => \$options{'help'}
	);

sub print_usage() {
	print "List all files :\n\t$0 -c list\n";
	print "Upload file :\n\t$0 -c upload -f file.txt\n";
	print "Print file :\n\t$0 -c view -f file.txt\n\n";
}

sub get_ftp() {
	my $ftp = Net::FTP->new($HOST, Port => $PORT, Debug => 0)
		or die "Cannot connect to $HOST: $@";
	$ftp->login() or die "Cannot login ", $ftp->message;
}

# ls / on remote ftp
sub list() {
	my $ftp = get_ftp();
	foreach my $f ($ftp->ls()) { print "$f\n"; }
	$ftp->quit;
}

# Upload a file
sub upload($) {
	my $file = shift;
	(-e $file) or return 1;

	my $ftp = get_ftp();
	$ftp->login() or die "Cannot login ", $ftp->message;
	$ftp->put("$file") or die "Cannot put $file ", $ftp->message;
	$ftp->quit;
}

# Read a file
sub view($) {
	my $file = shift;

	my $ftp = get_ftp();
	$ftp->get("$file") or die "Cannot read $file ", $ftp->message;
        if(-e $file) { print read_file($file); }
	$ftp->quit;
}

if($options{'action'} eq 'list') {
	list();
} elsif($options{'action'} eq 'upload') {
	upload($options{'file'});
} elsif($options{'action'} eq 'view') {
	view($options{'file'});
} else {
	print_usage();
}

More about design and security

This thin wrapper can be extended to do more tasks, such as checking allowed or disallowed name patterns or tidying files depending the uploader or the prefix in the name of the file. Remember, this is only on the client side! If you want real guarantees you would better have to implement some kind of protections on the server side too. But, the goal was not to discuss security here but to play with FTP! And I hope you had a pleasant tour with me and Net::FTP!

Listen to Larry Wall's State of the Onion 2000 on YouTube

Fri, 10 Jul 2020 09:07:55 +0000

Todd Rinaldo has uploaded Larry’s State of the Onion talk from YAPC ~~19100~~ 2000 to YouTube. Previously it was available in separate mp3 files, but this is much more accessible.

It’s a typical Larry talk filled with quirky, humorous observations about life and programming, and notably he announces the Perl 6 project.

Unfortunately the audio is low quality (hey it was 20 years ago at a low-budget conference); you can read a transcript of the talk here (with mp3 download links at the bottom). We also have collection of attendees’ reports from the conference.

If you enjoy Larry’s talks, you might like some of his other transcribed talks / articles.

Announcing Perl 7

Wed, 24 Jun 2020 11:34:35 +0000

image credit: Darren Wood, “7”, on Flickr.

This morning at The Perl Conference in the Cloud, Sawyer X announced that Perl has a new plan moving forward. Work on Perl 7 is already underway, but it’s not going to be a huge change in code or syntax. It’s Perl 5 with modern defaults and it sets the stage for bigger changes later. My latest book Preparing for Perl 7 goes into much more detail.

Perl 7 is going to be Perl 5.32, mostly

Perl 7.0 is going to be v5.32 but with different, saner, more modern defaults. You won’t have to enable most of the things you are already doing because they are enabled for you. The major version jump sets the boundary between how we have been doing things and what we can do in the future.

Remember, Perl was the “Do what I mean” language where the defaults were probably what you wanted to do. In Perl 4 and the early days of Perl 5, that was easy. But, it’s been a couple of decades and the world is more complicated now. We kept adding pragmas, but with Perl’s commitment to backward compatibility, we can’t change the default settings. Now we’re back to the old days of C where we have to include lots of boilerplate before we start doing something:

use utf8;
use strict;
use warnings;
use open qw(:std :utf8);
no feature qw(indirect);
use feature qw(signatures);
no warnings qw(experimental::signatures);

This is slightly better with v5.12 and later because we get strict for free by using setting a minimum version:

use v5.32;
use utf8;
use warnings;
use open qw(:std :utf8);
no feature qw(indirect);
use feature qw(signatures);
no warnings qw(experimental::signatures);

Perl 7 is a chance to make some of these the default even without specifying the version. Perl 5 still has Perl 5’s extreme backward compatibility behavior, but Perl 7 gets modern practice with minimal historical baggage. I’m personally hoping signatures makes the cut, but there’s still much to be done to make Unicode the default, so you’ll probably need to keep some of that:

use utf8;
use open qw(:std :utf8);

You might miss some seedier features that you shouldn’t be using anyway, such as the indirect object notation. Larry Wall said he had to do something for the C++ programmers:

my $cgi = new CGI;  # indirect object, but not in Perl 7
my $cgi = CGI->new; # direct object

But, the feature doesn’t really disappear in Perl 7. It’s already a setting in v5.32, but now with a different default.

What’s happening to Perl 5?

No one is taking Perl 5 away from you; it goes into long term maintenance mode—a lot longer than the two years of rolling support for the two latest user versions. That might be up to a decade from now (or half the time Perl 5 has already been around).

When is this happening?

The work is happening now, but you won’t need to worry about it for about six months when the first release candidate should appear. The target for a user release of Perl 7.0 within the next year, with some release candidates along the way.

This is an easy promise to keep, too, since Perl 7 is mostly v5.32 with different defaults. There’s no big rewrite or new features, although some currently experimental features may stabilize (please choose signatures!).

What about CPAN?

The Comprehensive Perl Archive Network, CPAN, has almost 200,000 modules. The maintained modules that people are using should still work, and for the rest there will be a compatibility mode. Remember Perl 7 is mostly v5.32 so you shouldn’t need to change much.

You may not know that the Perl5 Porters tests new versions against almost all of CPAN. There’s a long history of tools to check the effect that changes may have on the Perl community. As a module author, I often get messages from various people, mostly Andreas Koenig or Slaven Rezić, about weird things in my modules that may break with new Perls. Usually, it’s something I need to update anyway. Tracking down problems with existing code is a solved problem. Fixing code shouldn’t be that onerous because it’s still Perl 5, but with better practices.

Will there be a separate CPAN for Perl 7? No one has said there can’t be, but in the jump to Perl 7, the developers don’t want to redo what’s already working. This change should be manageable with as few side quests as possible.

Also, PAUSE, the Perl Authors Upload Server, has received quite a bit of love in the past couple of years. That makes it easier for them to adapt to future needs. The people working on that are experienced and talented, and they’ve made the codebase much more tractable.

Why the jump to a major version?

A major version can have a different contract with the user. A major version jump changes that contract with new default behavior, even if that conflicts with the past. There will be a way to reset all of those settings to the old Perl 5 default if you like. Perl 7 code will still be v5.32 code (mostly) in syntax and behavior though.

Sawyer speaks about three major market segments of Perl users:

People who are never going to change their code
People who use new features
People starting from scratch

Perl 5’s social contract is extreme backward compatibility, and has been amazingly successful with that. The problem is that the extreme backward compatibility works for those who won’t update their code, but doesn’t help the two other segments. The new features crowd has to deal with a longer boilerplate section in every program, and newbies wonder why they have to include so much just to create a program so people on StackOverflow won’t hector them over missing pragmas.

Why 7 and not 6?

There are two parts to this answer. First, “Perl 6” was already taken by what is now known as Raku. A long time ago, we thought that a very ambitious rewrite effort would replace v5.8. In short, that’s not what happened and the language has gone on to live a life of its own.

So, 7 was the next available number. That’s it. It’s just the next cardinal number in line. It’s not unheard of to make such a jump: PHP went directly from 5 to 7, and isn’t it time to steal something from that community? Consider these other weird jumps in history:

Solaris 2.6 to Solaris 7
Java 1.4 to Java 5
Postgres 9.x as the major version to Postgres 10 as the major version
Windows 3.1 to Windows 95 (98, ME, 2000, XP, Vista, 7, 8, 10)
TeX (each new version more closely approximates π)

At least it’s not Perl 34.

What’s disappearing?

Not much. Some things will be disabled by default, but again, this is essentially Perl 5.32 with the knobs and dials in different places. There are some things you should learn to live without, even in Perl 5 land. These are the likely candidates for the first round of changes:

indirect object notation
bareword filehandles (except maybe the standard filehandles)
fake multidimensional arrays and hashes (old Perl 4 trick)
Old-style prototype definitions (use :prototype() instead)

What’s appearing?

Not much. Perl 7 is mostly Perl v5.32, but with all of the features enabled by default. You don’t have to do anything to get most new features, such as postfix dereferencing, the new isa operator, or several other features. That’s the benefit of the new social contract a major version provides. It’s a hard boundary where new features can exist by default on one side without disturbing the other side.

What should I do right now?

If you need an older Perl to run your code, you are going to be fine. Those old versions are not going to disappear. Just like Perl 5.6 is still available, if that’s the version you wish to run.

If your code runs without a problem under strictures and warnings, and you are using modern Perl style, you’re probably mostly good. If you have some bareword filehandles, start converting those. Same with the indirect object notation.

With messy code, you aren’t out of luck. There will be compatibility modes to assist you in the transition from Perl 5 to 7 (but not Perl 5 to 8). A pragma will set the knobs and dials back to the old settings (but this is more of a one version thing):

use compat::perl5;  # act like Perl 5's defaults

For modules, there are some issues to shake out, but there will be a compatibility mechanism for those too.

The good news is that these things are already being tested by major Perl stakeholders in production settings. This isn’t a paper plan: it’s already happening and the rough edges are being sanded out.

And, v5.32 has one of these knobs and dials in place already. You can turn off the indirect object notation:

no feature qw(indirect);

But expect two more knobs or dials, maybe like:

no multidimensional;
no bareword::filehandle;

I’m collecting all of this information in Preparing for Perl 7, my latest offering through Perl School and LeanPub.

The bottom line

Perl 7 is v5.32 with different settings. Your code should work if it’s not a mess. Expect a user release within a year.

What's new on CPAN - May 2020

Sat, 20 Jun 2020 13:40:05 +0000

Welcome to “What’s new on CPAN”, a curated look at last month’s new CPAN uploads for your reading and programming pleasure. Enjoy!

APIs & Apps

Perl Layout Executor with App::plx
FIDO::Raw provides bindings to the libfido2 library
Geo::LibProj::cs2cs is a wrapper for the cs2cs command line client, part of the PROJ coordinate transformation library
Graph::Nauty provides bindings to Nauty (No AUTomorphisms, Yes?)
Post OCR requests to ocr.space’s API with OCR::OcrSpace

Config & Devops

Create reciples to declare and resolve dependencies between things with Beam::Make
Elliptic Curve Cryptography Library with Crypto::ECC
HealthCheck::Diagnostic::Redis provides a healthcheck for Redis
HealthCheck::Diagnostic::SMTP performs a connectivity healthcheck to an SMTP mail server
Show diffs of changes to files managed by Rex with Rex::Hook::File::Diff
Mask secrets in log files with String::Secret

Data

File::Groups returns file extensions and media types for different files
Get Digi-ID implementation with DigiByte::DigiID
Get elevation data for a given lat/lon using Geo::Elevation::HGT
JSON::Karabiner can easy JSON code generaation for Karabiner-Elements, the macOS keyboard customizer
Validate JSON against a schema against the latest draft with JSON::Schema::Draft201909

Development & Version Control

Documentation and tools for using Platypus with Go: FFI::Platypus::Lang::Go
Future::IO::Impl::Glib implement Future::IO using Glib
Mu::Tiny is an even tinier object system
Get a Try-Catch block (uses PPI) via Nice::Try (great name!)
Number::Textify turns numbers into human-readable strings (customizable)
Write composable, reusable tests with roles and Moo using Test2::Roo

Hardware

Device::Chip::BNO055 provides a chip driver for BNO055
PINE64::GPIO provides an interface to PineA64 and PineA64+ GPIO pins

Web

Catalyst::View::MojoTemplate: use Mojolicious templates in Catalyst views
Store Dancer2 session data in serealized files using Dancer2::Session::Sereal
Find elements in a HTML::Element DOM using CSS selectors with HTML::Selector::Element
Place a limit on “concurrent” promises with Mojo::Promise::Limiter
Add role-based access with context to a Mojo app via Mojolicious::Plugin::ContextAuth
POE::Component::SmokeBox::Recent::HTTP is an extremely minimal HTTP client

The Perl Ambassador: Gabor Szabo

Mon, 15 Jun 2020 07:30:00 +0000

This is the launch interview of a monthly series of interviews I’ll publish on perl.com. I can promise you, fun and entertaining interviews every month. So please watch this space. If you’d like me to interview you, or know someone you’d like me to interview, let me know. Take the same set of questions and send me your answers!

Gabor Szabo is a long time Perl developer and DevOps trainer and the author of the Perl tutorial and of Perl Maven and on Code Maven. He received a White Camel Award in 2008. He teaches training courses in Israel and around the world. He wears the hat of the chief editor of Perl Weekly newsletter, and is always happy to receive notable Perl news items for inclusion in its next issue.

How did you first start using Perl?

I was working at a start-up company near Jerusalem that had an AI-based software product which cost 1,000,000 USD with an additional 1-2 million cost for integration. That made sales a bit difficult so we were working on a related product that would sell for a mere 30,000 USD. Just to get our feet in the door of the potential buyers of our flagship product. We were using Scheme and AWK to write our compiler on Window 3.11. It was great fun, but I was also interested in all the sysadmin work in the company (we had a Novell NetWare 3.11 network and all the other areas that was not that interesting for the regular programmers. For example our build system.

Then Windows NT was introduced in 1993 and I got the opportunity to start setting it up. At around the same time our office was bought by NetManage, one of the pioneers in TCP/IP for MS Windows, in what is today called an Acqui-hiring. There we used some Rational ClearCase tools for bug tracking.

That’s the time when I first started to use Perl, probably in 1993 or 1994. I built an in-house web application to allow the developers to initiate a build of the software they were writing and to get notification when the build was ready. As I recall it was running on Windows. I also dealt with some of the bug-tracking automation that was running on some Unix system.

Which Perl modules are you constantly using? How do they make your life easier?

I hardly have any Perl-related work these days so I can’t really say, but when I need a script here and there I often use Capture::Tiny and keep re-creating it (partially and badly) in other languages. I love testing so anything that starts with Test:: is usually interesting to me.

I also run the Perl Maven and Code Maven sites on a Dancer-based application I wrote ages ago. The source is open though probably not very useful to anyone besides me. So I use Dancer2, Template Toolkit, DateTime, and DateTime::Tiny just to name a few. Oh and of course I love Perl::Critic and Devel::Cover.

Which Perl feature do you overuse?

I am not sure if any. Well, maybe except of Perl::Critic. I just noticed that I configured it that it won’t allow double-quotes around a string if there is nothing to interpolate in it. So “perl” is bad, ‘perl’ is good. I need to relax this.

I think I hardly ever used the fun features of Perl. I almost never use the do_this and do_that construct except for the open or die. I hardly use $_ and I think never use it explicitly. Maybe I was never a real Perl programmer :)

Which Perl feature do you wish you could use more?

Given that I hardly write Perl these days, any feature would be ok with me :)

I write mostly Python, Groovy, and recently Golang. So I would say I miss the autovivification, though definitely not the bug Perl has in being overenthusiastic about autovivification. I miss the possibility to move around some code, or comment out some code and try the rest without re-indenting everything. Though I don’t miss the time when I was begging my Perl students to indent their code.

I miss the CPAN Testers. (in these other languages). (And I do still encounter Perl code in some corporation written by people who have been writing Perl for 5-10 years and I always wonder why don’t they actually learn Perl…)

What one thing you’d like to change about Perl?

The community. Whatever that means. I wish people were prouder of their work and embraced the 21st century.

I wish they were more public about their work (e.g. announcements of releases of perl IMHO are only published on the p5porters list. Not on blogs.perl.org, not on perl.org, and not on https://news.perlfoundation.org/).

Very few module authors write about new releases of their code. There are very few people who write about Perl-related subjects. In many cases those posts are not linking to each other.

Some of the bloggers seem to have forgotten (or never learned) that links are a form of supporting each other. This is sad as it gives the impression that no-one uses Perl. Of course this is not new, it is just getting worse every year as the people who used to write about Perl stopped doing so or are writing about other subjects now.

It also makes it much harder to fill Perl Weekly newsletter. If it wasn’t for the Perl Weekly Challenge we would have half the size.

What is the future of Perl?

I think the number of “Perl programmers” will continue to decline and with that less and less problems will have a solution on CPAN.

However, the more interesting thing to me is what happens to the “Perl programmers”. The same way the impact of Larry Wall on programming goes far beyond the people who use (or ever used) Perl. Or patch for that matter.

For example I would love to know what happened to all the people who received a White Camel Award even if they don’t write Perl any more.

Same with the most high-profile CPAN authors. Even if they don’t write Perl any more, their thinking was formed by Perl and by all the nice things the community had throughout the last 32+ years.

Perl Hacks, Perl School, and the future of Perl publishing

Thu, 28 May 2020 09:44:03 +0000

Dave Cross, long-time Perl user, trainer, and author, recently released The Best of Perl Hacks, a curated collection of his best posts from his Perl Hacks blog. His imprint, Perl School, has published six e-books, including two that I wrote.

There’s an unrelated book, Perl Hacks: Tips & Tools For Programming, Debugging, And Surviving, by chromatic, Damian Conway, and Curtis “Ovid” Poe. It’s also very good, but completely separate from Dave’s.

What is Perl Hacks? When did you start it and what do you like to post there?

Perl Hacks is my Perl blog. It’s where I post all my Perl-related articles.

I started it in May 2009. Before then, pretty much everyone in the Perl community used to blog on a site called Use Perl. But that site was starting to look a bit dated and a number of people moved their blogs to other places at around the same time. It’s no coincidence that the blogs.perl.org site also dates from the same year.

My rule for choosing what to post on the site is basically “is this about Perl?” But looking back over the lifetime of the site (which I did when compiling the book) I noticed that the type of article had changed over time. When I first started, there were a lot of “newsy” entries—”London.pm will be holding these meetings”, “I’m running a training course”, things like that. But later I started posting longer articles about the Perl community or interesting technical corners of Perl.

How did you choose what went into the ebook? What are some of your favorite posts? Which ones got the best responses?

I basically read through the site over a weekend. I ignored all of the short articles and anything that was topical and would no longer be interesting. I then did a second pass, planning to get to about fifty articles. I think I ended up with fifty-seven.

There are a couple of technical articles that I’m particularly pleased with. Dots and Perl explains Perl’s five operators that are just made of dots. Can you name them all? Ok, strictly speaking, one of them isn’t actually an operator. And Subroutines and Ampersands was written so I had somewhere to point people who still insist on putting ampersands on subroutine calls. It carefully explains why it’s rarely necessary (and hasn’t been since Perl 5 was released).

There’s also The Long Death of CGI.pm which investigated the effects that removing CGI.pm from the Perl core distribution would have. I thought there was some nice detective work in there.

The post that got the most response was called You Must Hate Version Control Systems. The title was taken from a Perl job ad from a company that no-one seemed to want to work for. The person who posted the ad turned up in the comments and tried to explain why he wrote that, but I don’t think anyone was convinced. This was the one time that I regretted that I couldn’t use a post’s comments in the book.

What is Perl School? Why did you start it? What was the first book?

Perl School was a brand that I started to use in 2012 for some training I ran in London. I had theory that people weren’t keeping their Perl knowledge up to date and many employers weren’t keen on investing in training about what was “just a scripting language.” So, I reasoned, if I ran low-cost training courses at the weekend, people would come on them and the average level of Perl knowledge in London would rise.

I ran these courses for a year before putting them on ice. But I liked the brand and knew that I wanted to use it again in the future.

In 2015 I wrote a beginners’ Perl tutorial for Udemy. They published it on their site and I often pointed people at it. But a couple of years later, I checked their site to find that a CSS upgrade had rendered the page pretty much unreadable. I pointed this out to them and after a bit of discussion, they told me that they weren’t going to invest the time to fix it but said that I was welcome to publish it elsewhere.

At the same time I had been experimenting with building ebooks from Markdown and had developed the pipeline that I still use today. So the first Perl School book was the serendipitous meeting between my experimental ebook pipeline and a long piece of text that I wanted to get out to as many people as possible.

It’s called Perl Taster: Your First Two Hours With Perl. I published it just before the 2017 London Perl Workshop and announced it in a lighting talk at that workshop. I invited other people to contribute books, offering to help with the technical parts of getting them published. John Davies approached me about the book that became Selenium and Perl and it’s just grown from there.

What was your experience giving low-cost and free Perl training in the UK?

I’ve run Perl training at all levels in the UK. Five years ago I’d still get two or three enquiries a year from companies who were interested in training, but that’s all dried up. I can’t remember the last time someone asked me about running a course for them.

The Perl School courses were popular for a while. I’d get twenty or so people giving up their Saturday and paying a small fee to get Perl training. But after a year, the interest started to wane. I cancelled the last one because I didn’t get enough attendees to make it worthwhile.

For many years I’ve run a free ninety-minute or two-hour course as part of the London Perl Workshop. I always get a pretty good turn-out for those. But, to be honest, that’s about the only place I can guarantee much interest in Perl training in the UK these days.

You previously wrote the print books Data Munging with Perl (Manning Press) and Template Toolkit (O’Reilly & Associates, with Andy Wardley and Darren Chamberlain). How is Perl School’s process different than what you experienced with those publishers?

I guess the main difference is that there’s a lot less process involved with the Perl School books.

With a traditional publisher, there are lots of departments involved. The editor will want to know when the manuscript will be ready because they will want to book time from technical reviewers and proof-readers. They’ll also need to plan in designers and even book printing time on the presses. All of that means there’s a lot of pressure on the author to make a plan for getting the book written and then to stick to that plan.

With an ebook, it’s all a lot less structured. I largely rely on authors to arrange their own technical reviewers. I’ll do a bit of proof-reading. And we haven’t (as yet) used any designers—that probably shows, to be honest.

So I’m not going to pressure an author to finish a manuscript. When you’re ready, I’ll steer you through turning your Markdown into an ebook and then publishing it on Amazon. If it takes longer than you expected, then so what?

In many ways, I see parallels with the Lean Startup ideas of Eric Ries. We’re small and we’re agile. If you come to me with a completed book, we might well be able to get it on Amazon in a week or two. For a traditional publisher, that time will be months.

What do you think are the biggest challenges to technical publishing today? How does something like Perl School respond to that?

Traditional publishing is an expensive business. Publishers need to make a lot of money just to break even on a book. I don’t know the details, but they have to sell a certain number of copies in order to make it worth publishing a book. And that, in turn, means that they will rarely take a risk. For a technical publisher, that means only publishing books about technologies that have reached a certain level of usage.

People are also buying fewer technical books. Technologies change quickly and many books will be out of date before they get to the bookshops. If you want up to date information about your favourite technologies then you’re probably better off going to the developer’s web site.

For a publisher like Perl School, the economics are different. We have far smaller costs and (as I mentioned before) we can get books in customers’ hands far more quickly.

Large technical publishers have largely abandoned Perl. They just don’t see that they would get the level of sales needed to justify a new Perl book. Perl School is happy to take that risk—because, really, it’s a tiny risk for us.

What are your personal reading preferences? Which device do you like, what size screen do you need, and which format works best for you?

I do like a real book. But they take up too much space, so I’ve pretty much completely stopped buying them over the last five years. I like being able to access all of the ebook part of my library from a device that I can carry in my pocket. One thing that would make me really happy is a device that could rip my existing paper library to ebooks in the same way that we all ripped our CDs to MP3s.

Currently, I read ebooks on an 8” Amazon Kindle Fire. That’s just bigger than a paperback and fits in a (large) pocket in the same way that a book would. I also have a 10” Pixel Slate which I’ll often use for reading in my house. I’m rather firmly locked into the Amazon ebook ecosystem, so I prefer to find books in Mobi format—but I can drive Calibre, so I’m happy to convert from other formats.

I get mildly annoyed by web sites that promise me an ebook and then deliver a PDF. PDFs are made to be read at a certain size and if you’re reading them on a smaller screen it will either be too small to read or you’ll need to scroll back and forth a lot. A proper ebook format (like Mobi or ePub) will reformat pages for any combination of screen size and font size.

How can someone write a book for Perl School? What topics do you think would be most interesting?

Simply email me at hello@perlschool.com with your suggestions. I’ll talk you though the technical side of getting the book written and published on Amazon—it’s really not very hard.

I’d pretty much consider any Perl-related topic. I would never have thought of publishing books on Selenium or Cucumber, but John Davies wrote them and I’ve published them. And they seem to be selling.

But there are books I’d like to see. I’d love to publish books on the various “Modern Perl” tools that we all use—Moose, DBIx::Class, things like that—and all of Perl’s popular web frameworks.

I think the one I’d most like to see is one that, in my head, is called “Modern Core Perl”. It covers all of the important changes in the core Perl language back to version 5.10. Basically, it’s a tutorial based on all of the perldeltas. I’ve thought about writing it myself a few times, but I just don’t have the time.

What's new on CPAN - April 2020

Wed, 20 May 2020 01:53:33 +0000

Welcome to “What’s new on CPAN”, a curated look at last month’s new CPAN uploads for your reading and programming pleasure. Enjoy!

APIs & Apps

Use the open, decentralizedc Matrix communication network with Net::Matrix::Webhook
Call Docker CLI commands from Perl using Docker::CLI::Wrapper
InfluxDB::Client::Simple is a lightweight InfluxDB client
Termux::API provides a Perly interface to the popular Android terminal emulator
Get financial data via Yahoo Finance with Yahoo::Finance

Config & Devops

Grant Street Group added more healthcheck modules:
- HealthCheck::Diagnostic::DBHCheck checks a database handle has read/write permissions
- HealthCheck::Diagnostic::SFTP checks secure FTP access
- HealthCheck::Diagnostic::WebRequest checks HTTP/HTTPS connectivity

Data

Moo-ify DBIx::Class rows using DBIx::Class::Moo::ResultClass
Data::Random::Structure::UTF8 can fill a data structure with random UTF-8 data
JSON::Schema::Generate generates JSON schemas from data structures
Use named (instead of positional) placeholders with SQL queries via SQL::Bind

Development & Version Control

Docker::Names::Random generates random strings like Docker does for container names (e.g. “lazy_fermat”)
A class based approach for scripting options: Getopt::Class
Get simple, non-blocking IPC with IPC::Simple
MooseX::amine ++ for module naming, it let’s you examine the methods and properties of Moose objects
Return from multiple blocks in one go with Return::Deep
Test::Ability provides property-based testing routines
fs::Promises provides a promises interface to non-blocking file system operations

Web

Manage passwords in Dancer2 with Argon2 using Dancer2::Plugin::Argon2
Use the Minion job queue in your Dancer2 apps with Dancer2::Plugin::Minion
Override any method in your Kelp application with KelpX::Hooks
Mojo::Log::Role::Color adds color to your interactive mojo logs
Mojo::UserAgent::Role::Signature automatically signs request transactions
Multipart::Encoder is an encoder for mime-type multipart/form-data.

What's new on CPAN - March 2020

Wed, 29 Apr 2020 01:05:57 +0000

Welcome to “What’s new on CPAN”, a curated look at last month’s new CPAN uploads for your reading and programming pleasure. Enjoy!

APIs & Apps

Get a colorful calendar in the terminal with App::week
Google::reCAPTCHA::v3 is another Google captcha module
GraphQL::Client … is a GraphQL client
Masscan::Scanner makes it easy to use the masscan port scanner.
Make non-blocking requests to LastFM with Mojo::WebService::LastFM
Use Telegram’s Bot API with Net::API::Telegram
Net::OpenVAS let’s you program Greenbone’s OpenVAS platform with Perl
Use Elasticsearch 6.x APIs with Search::Elasticsearch::Client::6_0

Config & Devops

Block::NJH is interesting; add it to your CPAN distribution to “prevent your tests from running on NJH’s broken smokers”
Config::Structured provides “generalized and structured configuration value access”
Ping a database handle to check its health using HealthCheck::Diagnostic::DBHPing
HealthCheck::Diagnostic::FilePermissions checks filepaths for expected permissions

Data

DB::Object is a database abstraction built on top of DBI
Inspect DBIC objects in a compact format using Data::Tersify::Plugin::DBIx::Class
MIME::Base32::XS is a faster Base32 encoder/decoder
Statistics::Covid fetches and manages Covid-19 statistics
Get a Postgresql mock server for testing via Test::PostgreSQL::Docker
Bind Perl data structures into XML with XML::BindData
XML::Minifier is a configurable XML minifier

Development & Version Control

Devel::Wherefore helps debug Perl: “Where the heck did these subroutines come from?”
Create relative symbolic links File::Symlink::Relative
Module::Generic is another class library, it uses AUTOLOAD for getter/setter methods
new saves you from typing module names twice in one liners

Science & Mathematics

Bio::DB::Taxonomy::sqlite stores and manages NCBI data using sqlite
Math::Polynomial::Chebyshev creates Chebyshev polynomials

Web

Store Catalyst sessions in Redis with Catalyst::Plugin::Session::Store::RedisFast
Not a traditional distribution, but Mojo::Server::AWSLambda contains a simple example of how to define an AWS Lambda function which uses Mojo
Mojolicious::Plugin::Sticker combines Mojo apps into a single app

Observing Coronavirus Pandemic with Raku

Thu, 02 Apr 2020 08:45:00 +0000

Every few years a new unknown virus pops up and starts spreading around the globe. This year, the situation with COVID-19 is different not only because of the nature of the virus but also because of the Internet. Whilst we have instant access to new information (which is often alarmist in tone) we also have the ability to access data for ourselves.

Johns Hopkins University Center for Systems Science and Engineering synthesizes COVID-19 data from different sources, and displays it on their online dashboard. They also publish daily updates in CSV files on GitHub.

I decided to ingest their CSV data and display it using different visualizations to reduce panic and provide a way to quickly see real numbers and trends. The result is the website covid.observer. The source files are available in the GitHub repository.

For years Perl has been known for BioPerl. Let’s see what Raku can bring to society as its great at manipulating text data. The heart of the site is a Raku program and a few modules that parse data and create static HTML pages.

I’m going to show you a few of the most useful features that Raku offers to developers.

The `MAIN` function

The program works in three modes: parsing population data, getting updates from the COVID raw data, and generating HTML files. Raku gives us a very handy way to process command line arguments by defining different variants of the MAIN function. Each variant is mapped to different command line parameters, and Raku automatically dispatches to the matched variant, which helps me to run the program in the desired mode.

Here are the variants:

multi sub MAIN('population') {
    . . .
}

multi sub MAIN('fetch') {
    . . .
}

multi sub MAIN('generate') {
    . . .
}

We don’t need to parse the command-line options ourselves, nor use any modules such as Getopt::Long to do it for us. Moreover, Raku emits “usage” help text if the program is run with incorrect or missing arguments:

$ ./covid.raku
Usage:
  ./covid.raku population
  ./covid.raku fetch
  ./covid.raku generate

Salve J. Nilsen proposed to add another MAIN function that prints the SQL commands for initializing the database. This example shows how how to define Boolean flags for command line options:

multi sub MAIN('setup', Bool :$force=False, Bool :$verbose=False) {
    . . .
}

Note the : before the parameter name. We’ll see it again later.

An additional POD comment can be added before each version of the function to print a better usage description, for example:

#| Fetch the latest data and rebuild the database
multi sub MAIN('fetch') {
    . . .
}

Now the program prints a more helpful usage message:

Usage:
  ./covid.raku population -- Parse population CSV files
  ./covid.raku fetch -- Fetch the latest data and rebuild the database
  ./covid.raku generate -- Generate the website

Reduction operators

Reduction operators are really useful. Let me remind you what a reduction operator is. It’s actually a meta-operator: an infix operator surrounded by square brackets.

In the program the reduction operator is widely used for computing totals across the data sets (e.g. for the World, or across Chinese provinces). Let us examine a few cases of increasing complexity:

First there’s a simple hash, and we need to add up its values:

my %data =
    IT => 59_138,
    CN => 81_397,
    ES => 28_768;

my $total = [+] %data.values;
say $total; # 169303

This is the classic use case for the reduction operator. What I noticed during the work is that the [-] construct helps when you need to reduce some value by a few other values:

my %data =
    confirmed => 100,
    failed    => 1,
    recovered => 70;

my $active = [-] %data<confirmed recovered failed>;
say "$active active cases";

Using a hash slice in the form of %h<a b c> also helps to make the code more compact. Compare this with the straightforward approach:

my $active = %data<confirmed> - %data<failed> - %data<recovered>;

Filtering data

For our second case, the hash values are not scalars but hashes themselves. The >> hyperoperator can be used to extract deeply located data. Let me demonstrate this on a simplified data fragment:

my %data =
    IT => {
        confirmed => 59_138,
        population => 61, # millions
    },
    CN => {
        confirmed => 81_397,
        population => 1434
    },
    ES => {
        confirmed => 28_768,
        population => 47,
    };

my $total = [+] %data.values>><confirmed>;
say $total; # 169303

An alternative and cleaner way is using the map method to access the data:

my $total2 = [+] %data.values.map: *<confirmed>;
say $total2; # 169303

Finally, to exclude a country from the results, you can grep the keys in-place:

my %x = %data.grep: *.key ne 'CN';
my $excl2 = [+] %x.values.map: *<confirmed>;
say $excl2; # 87906

Notice that calling the hash’s grep method is much handier than trying to loop over the keys and filter them:

my $excluding-china =
    [+] %data{%data.keys.grep: * ne 'CN'}.values.map: *<confirmed>;
say $excluding-china; # 87906

Hyper operators

In the previous section I showed how to apply the same action to each element of a list using >>. Now let us take a look at a real example of how I used the hyper operator >>->> to compute the deltas of number series:

my @confirmed = 10, 20, 40, 70, 150;
my @delta = @confirmed[1..*] >>->> @confirmed;
say @delta; # [10 20 30 80]

The array contains a series of values for the given period of time. The task is to compute how many new cases happen in each day. Instead of using a loop, it is possible to simply ‘subtract‘ an array from itself but shifted by one element.

The >>->> operator takes two data series: the slice @confirmed[1..*] of the original data without the first element, and the original @confirmed array. For a given binary operator (- in this example), you can construct four hyper operators: >>-<<, >>->>, <<->>, <<-<<. The chosen form allows us to ignore the extra item at the end of @confirmed when it is applied against @confirmed[1..*].

Junctions

Let me demonstrate a way of using the junction operator | which I discovered recently. It chooses the ending for the given ordinal number:

for 1..31 -> $day {
    my $ending = do given $day {
        when 1|21|31 {'st'}
        when 2|22    {'nd'}
        when 3|23    {'rd'}
        default      {'th'}
    }

    say "$day$ending";
}

The when blocks catch the corresponding numbers that need special endings. Junctions such as 1|21|31 are more elegant than a regular expression or a chain of comparisons.

Optional and named parameters

Parameter processing is simple in Raku. This function accepts two positional hash parameters:

sub chart-daily(%countries, %totals) {
    . . .
}

I can easily add optional named parameters:

sub chart-daily(%countries, %totals, :$cc?, :$cont?, :$exclude?) {
   . . .
}

A colon before the name makes the parameter named, and the question mark makes it optional. I am using this to modify the behavior of the same statistical function for aggregating data over the whole World, the continents, or to exclude a single country or a region:

Generating data for the whole World:

chart-daily(%countries, %per-day);

For a single country:

my $cc = 'CN';
chart-daily(%countries, %per-day, :$cc);

For a continent:

my $cont = 'Europe';
chart-daily(%countries, %per-day, :$cont);

World data excluding China:

chart-daily(%countries, %per-day, exclude => 'CN');

Getting data for China without its most affected province:

chart-daily(%countries, %per-day, cc => 'CN', exclude => 'CN/HB');

A built-in template engine

The project generates more than 200 HTML files, so templating is an important part of it. Fortunately Raku has a great out-of-the-box templating mechanism, which is much more powerful than simple variable interpolation.

A minimal example is substituting variables:

return qq:to/HTML/;
    <div id="countries-list">
        $html
    </div>
    HTML

By the way, notice that Raku lets you keep the indentation of a multi-line string by simply indenting its closing symbol. No extra spaces at the beginning of the lines will appear in the result.

A more exciting thing is that you can embed Raku code blocks into strings, and those blocks can contain any logic you need to make a right decision somewhere in the middle of the template:

my $content = qq:to/HTML/;
    <h1>Coronavirus in {$country-name}</h1>

    <div class="affected">
        {
            if $chart2data<confirmed> {
                'Affected 1 of every ' ~
                    fmtnum((1_000_000 * $population /
                        $chart2data<confirmed>).round())
            }
            else {
                'Nobody affected'
            }
        }
    </div>
    HTML

Here, the string builds itself depending on data. For each generated country, the string ’chooses‘ which phrase to embed and how to format the number. The if block is a relatively big chunk of Raku code that generates a string, which is used in place of the whole block in curly braces. Thus, inside this embedded code block you can freely manipulate data from the outside code.

Afterword

I must say that it is quite exciting to use Raku for a real project. As you can see from the examples, many of its ‘strange‘ features demonstrate how useful they are in different circumstances. Examine the code in the GitHub repository and follow the updates about the site on my blog.

Articles on Perl.com - programming news, code and culture

Bang Bang

Perl is nice

I have something for you

Plenty is no plague

A magic incantation

-x is fun

startperl

Resources

Stupid DATA Tricks

This is the end, my friend, the END

Some other DATA tricks

Program size

Read it twice

Using line numbers

Multiple embedded files

The Perl Ambassador: Damian Conway

When and how did you get started with Perl?

What made you write Perl Best Practices?

Do you follow your own suggestions in the book?

What do you think about “Perl 7” currently being discussed widely?

Do you follow the development of “Cor”?

You have loads of handy modules published on CPAN. What are your top 5 contributions and why?

Lingua::EN::Inflexion

Parse::RecDescent

Quantum::Superpositions

Regexp::Debugger

PPR

You visit Europe regularly and give lectures on Perl/Raku related topics. Do you have any memorable story to share?

You briefly blogged about the weekly challenge, how did you find about it?

And what was your experience with it?

Where do you get the ideas for your next CPAN contributions?

What would you suggest to someone starting Perl?

I noticed you are associated with Raku these days. Do you still use Perl?

Searching Internet RFCs

What are RFCs?

How it works

Other uses and suggestions

The Perl Ambassador: Curtis 'Ovid' Poe

How did you first start using Perl?

What do you think about Perl 7? Do you see the Perl is on the path of recovery?

What inspired you to start the project “Cor”? When are you planning to release it?

How does your company “All around the world” help people with Perl?

How is “Tau Station” going and how much Perl helped in building the game?

Which Perl modules are you constantly using? How do they make your life easier?

Which Perl feature do you overuse?

Which Perl feature do you wish you could use more?

What one thing you’d like to change about Perl?

What's new on CPAN - June 2020

APIs & Apps

Config & Devops

Data

Development & Version Control

Hardware

Language & International

A tour with Net::FTP

A bit of context

Download and install ftpd

Test with ftp

Simple client in Perl

Simple listing

Upload

Put things together

More about design and security

Listen to Larry Wall's State of the Onion 2000 on YouTube

Announcing Perl 7

Perl 7 is going to be Perl 5.32, mostly

What’s happening to Perl 5?

When is this happening?

What about CPAN?

Why the jump to a major version?

Why 7 and not 6?

What’s disappearing?

What’s appearing?

What should I do right now?

The bottom line

What's new on CPAN - May 2020

APIs & Apps

Config & Devops

Data

The `MAIN` function