Perl makes writing Twilio telephony applications simple and enjoyable. My previous article, Automating Telephony with Perl and Twilio showed how to use WWW::Twilio::API to make a phone survey using Twilio's text-to-speech synthesizer as well as send an SMS message to a phone number of your choice.

This article builds an easy phone menu application using Twilio's TwiML application language, WWW::Twilio::TwiML, and Mojolicious.

Installation and Twilio account creation

To begin, install WWW::Twilio::TwiML and Mojolicious using your favorite method (see the previous article for more ideas). You'll need a Twilio account (it's free to create an account, plus Twilio gives new users US $30 for calls--plenty for several weeks of testing). The previous article covers installation and getting setup with a Twilio account in more detail. With WWW::Twilio::TwiML and Mojolicious installed and a Twilio account active, it's a great time to get a little more familiar with Twilio's Dashboard.

Twilio Dashboard

The Twilio Dashboard is where to find your Twilio sandbox information. The sandbox section of the dashboard is located below the fold of the dashboard page:

Twilio Dashboard

You should see the sandbox number and pin. You can set our inbound voice and inbound SMS handler URLs there (I'll explain those soon.) First, a primer on how Twilio works.

TwiML in the Twilio flow

Twilio's basic flow goes something like:

Handling an inbound call to Twilio
(This diagram is not an official Twilio diagram; it merely describes the author's mental model of how Twilio works and may differ wildly from Twilio's actual implementation.)

  1. The caller dials (or sends an SMS to) "555-867-5309" on their phone--this is your Twilio sandbox or purchased number. Twilio's inbound call dispatcher receives the call or text.

  2. Once the connection is made, the dispatcher makes an HTTP GET or POST to the Voice or SMS URL specified given for this number. Remember the Voice URL in the Sandbox App of the Twilio Dashboard shown above? That's the one. For purchased Twilio numbers, you set the voice and SMS URLs under the "Numbers" tab in the Dashboard.

  3. The Voice or SMS URL specified in the Sandbox App responds to the Twilio request with a TwiML document. Twilio's TwiML parser reads this document, then executes the "verbs" specified in the TwiML document. For example, if the TwiML document contained a <Say> verb, Twilio's text-to-speech synthesizer would "read" the text to the caller. If the document specified a <Dial> verb and number, Twilio would dial the number and connect the caller to it. <Redirect> verbs tell Twilio to fetch another TwiML document.

You may recall an example in the previous article which used Twilio's voicemail TwiML handler to conduct a brief phone survey. While you can sometimes manipulate third-party TwiML applications to do what you want, TwiML is so simple to use that you'll find it's often easier to write your own.

TwiML basics

TwiML is a subset of XML. Here is a TwiML document that when read by Twilio's parser, will say to the caller, "Foosball at 10 o'clock!" using the text-to-speech synthesizer:

<?xml version="1.0" encoding="UTF-8" ?>
<Response>
  <Say voice="man">Foosball at 10 o&apos;clock!</Say>
</Response>

Some TwiML verbs are nestable:

<?xml version="1.0" encoding="UTF-8" ?>
<Response>
  <Gather action="/menu.cgi" numDigits="1" method="GET">
    <Say voice="man">
      Ping pong at high noon!
      Hit 1 if you&apos;re with me.
      Hit 2 if you&apos;re a loser.
    </Say>
  </Gather>

  <Say>You must make a choice!</Say>
</Response>

The <Gather> verb tells Twilio to start gathering key presses. Meanwhile, Twilio will hand off the contents of the <Say> verb to the text-to-speech handler. Nesting the <Say> inside of the <Gather> lets the caller push a key anytime during the <Say> verb to interrupt it and process your choice. The final <Say> element only executes if the <Gather> fails (e.g., the caller doesn't press a key).

Twilio's excellent documentation details all of the available TwiML verbs. Armed with a little knowledge, you're dangerouly close to making something useful.

Simple phone menu

You've been asked to build a phone menu system for a young urban professional and his family. The application should accept an incoming call, prompt the caller with a numeric menu, and connect the caller with the number of their choice.

No problem. Start with a simple static TwiML document called menu.xml:

<?xml version="1.0" encoding="UTF-8" ?>
<Response>
  <Gather method="GET" action="/menu_handler" numDigits="1">
    <Say>Press 1 for Ryan</Say>
    <Say>Press 2 for Liz</Say>
    <Say>Press 3 for Jason</Say>
    <Say>Press 4 for Erin</Say>
    <Say>Press 5 for Rachel</Say>
    <Say>Press 6 for Gilligan</Say>
    <Say>Press 7 for Potsie</Say>
  </Gather>
</Response>

You can put this file on the web and update the URL in the "Voice URL" field of the Sandbox section of the Dashboard. Anyone who dials the sandbox number, will hear a Twilio's text-to-speech voice reading the menu options. Anybody remember the "S.A.M." speech synthesizer from the mid 80's (Commodore 64 or Atari 800)? You've come a long way, baby!

While this static TwiML file would work, it does mean you'll have two files to update when a phone number changes. Fix that by consolidating the /menu and /menu_handler "routes" (the Mojolicious word for URL handlers) into a single program to generate TwiML dynamically.

WWW::Twilio::TwiML, briefly

WWW::Twilio::TwiML is a special-purpose XML generator and supports several programming styles. For example, the following two code snippets create this TwiML document:

<?xml version="1.0" encoding="UTF-8" ?>
<Response>
    <Say voice="man">Kilroy was here</Say>
</Response>

Snippet number 1:

my $say = WWW::Twilio::TwiML->new();
$say−>name('Say');
$say−>content("Kilroy was here");
$say−>attributes({voice => "man"});

my $resp = WWW::Twilio::TwiML->new();
$resp−>name('Response');
$resp−>content($say);

my $tw = WWW::Twilio::TwiML->new();
$tw−>content($resp);
print $tw−>to_string;

And snippet number 2:

my $tw = WWW::Twilio::TwiML->new();
$tw−>Response−>Say({voice => "man"}, "Kilroy was here");
print $tw−>to_string;

The second snippet uses a technique called "method chaining"; if you've used the jQuery module for Javascript, you may already know how powerful chaining object methods can be in certain contexts. WWW::Twilio::TwiML makes chaining possible because each TwiML verb method is a constructor of another WWW::Twilio::TwiML object.

When you chain TwiML objects like this:

$tw->Response->Say("Eat at Joe&apos;");

The Response object is created as a child of the top $tw object. The Say object is created as the child of the Response object. When the $tw object's to_string method is invoked, like this:

print $tw->to_string;

WWW:Twilio::TwiML crawls down $tw's list of children, recursively invoking to_string until the last child. The whole process creates this output:

<?xml version="1.0" encoding="UTF-8" ?>
<Response>
    <Say>Eat at Joe&apos;</Say>
</Response>

It's may look a little foreign, but it makes for concise and readable TwiML.

Getting your mojo on

Now that you have a feel for creating TwiML documents, you need a way to serve them from the web.

Mojolicious is an easy-to-use web application framework. Just a few lines and you have a sweet little web app to serve TwiML. Feel free to read a little of the Mojolicious documentation. This author highly recommends going through the Mojolicious::Lite documentation first: almost everything you learn in Mojolicious::Lite also applies to the larger Mojolicious application framework.

To write just enough Mojolicious to do what you've already done with the static TwiML document, you need only two modules for this entire application. (Neither has any dependencies--you're welcome.) Keep in mind that Mojolicious::Lite enables 'warnings' and 'strict' by default: no sloppy programming allowed!

This application implements a Mojolicious handler for the /menu route which is only called when the application receives an HTTP GET to the /menu URI.

#!/usr/bin/env perl

use Mojolicious::Lite;
use WWW::Twilio::TwiML;

get '/menu' => sub {
    my $self = shift;

    $self->render(format => 'xml',
                  text   => <<'_TWIML_');
<?xml version="1.0" encoding="UTF-8" ?>
<Response>
  <Gather method="GET" action="/menu_handler" numDigits="1">
    <Say>Press 1 for Ryan</Say>
    <Say>Press 2 for Liz</Say>
    <Say>Press 3 for Jason</Say>
    <Say>Press 4 for Erin</Say>
    <Say>Press 5 for Rachel</Say>
    <Say>Press 6 for Gilligan</Say>
    <Say>Press 7 for Potsie</Say>
  </Gather>
</Response>
_TWIML_
};

app->start;

That's all it takes for a Mojolicious application (source here). Mojolicious packs a full stack HTTP 1.1 web server, making tests easy. In one shell, start your Mojolicious program as a daemon (use Ctrl-c to stop when finished):

$ perl menu_part daemon
[Mon Nov 14 21:41:34 2011] [info] Server listening (http://*:3000)
Server available at http://127.0.0.1:3000.

In another shell, pretend you're Twilio's application server and fetch the TwiML:

$ curl http://localhost:3000/menu
<?xml version="1.0" encoding="UTF-8" ?>
<Response>
  <Gather method="GET" action="/menu_handler" numDigits="1">
    <Say>Press 1 for Ryan</Say>
    <Say>Press 2 for Liz</Say>
    <Say>Press 3 for Jason</Say>
    <Say>Press 4 for Erin</Say>
    <Say>Press 5 for Rachel</Say>
    <Say>Press 6 for Gilligan</Say>
    <Say>Press 7 for Potsie</Say>
  </Gather>
</Response>

Is this progress? This is same TwiML as in menu.xml, but now it's created with Mojolicious. At this point you can upload your app to a publicly accessible server, and, assuming Mojolicious is also installed there, you can start your application. Then go to the Twilio Dashboard and replace the existing Voice URL with:

http://your.server.org:3000/menu

Important note about Mojolicious and HTTP GET

Mojolicious's get method only responds to HTTP GET method requests. Make sure that when you set the Voice URL in the Twilio Dashboard, you also change the HTTP method to GET. Alternatively, you could use Mojolicious's post method, which only responds to HTTP POST, or the any method, which accepts both GET and POST (as well as PUT and DELETE), but making Twilio use HTTP GET seems more appropriate.

Putting it all together

Now it's time to improve the program because it still serves static TwiML. It also needs a programmatic way tell Twilio what to do when the caller presses a key.

Start over and make a hash for the phone menu. This could be put into a separate file--and probably should be--but it wouldn't be a proper tutorial without something left as an exercise.

#!/usr/bin/env perl

use Mojolicious::Lite;
use WWW::Twilio::TwiML;

my %list = ( 1 => { name => 'Ryan',
                    number => '+19165557720' },
             2 => { name => 'Liz',
                    number => '+19165551211' },
             3 => { name => 'Jason',
                    number => '+19285550122' },
             4 => { name => 'Erin',
                    number => '+19285551729' },
             5 => { name => 'Rachel',
                    number => '+18015553992' },
             6 => { name => 'Gilligan',
                    recording => 'http://www.televisiontunes.com/'
                              .     'themesongs/Gilligans%20Island.mp3' },
             7 => { name => 'Potsie',
                    recording => 'http://www.televisiontunes.com/'
                              .     'themesongs/Happy%20Days%20-%20Season%202.mp3' },
           );

get '/menu' => sub {
    my $self = shift;

This should looks mostly familiar so far. Now create $msg which holds all of the text you want Twilio to "Say" to the caller:

    my $msg = join '. ',
      map { "Press $_ for $list{$_}->{name}" }
        sort keys %list;

It's time to build the TwiML document, and this could use some error handling in case the caller doesn't press a key. Finally, print everything via Mojolicious's render() method:

    my $tw   = WWW::Twilio::TwiML->new();
    my $resp = $tw->Response;
    $resp->Gather({action    => $self->url_for('/menu_handler'),
                   method    => 'GET',
                   numDigits => 1})
         ->Say({voice => 'woman'}, $msg);

    $resp->Say("You need to make a choice or hang up.");
    $resp->Redirect("/menu");

    $self->render(format => 'xml',
                  text   => $tw->to_string);
};

Remember to start Mojolicious's event loop; this code should always go at the end of your application, as it never returns:

app->start;

(Source here). If declarative-style programming is new to you, don't fret: under the hood, Mojolicious simply creates a map for itself that says "when I receive an HTTP GET for '/menu', I should execute this subroutine". The app->start routine begins a loop that waits for said request, then handles it as you've defined (declared) it.

The menu handler route

Most of the hard work is done now. As specified in the Gather's action attribute above, when the caller presses a key, Twilio will perform an HTTP GET on /menu_handler. Add one more Mojolicious route:

get '/menu_handler' => sub {
    my $self   = shift;
    my $choice = $self->param('Digits') || 0;

    my $tw     = WWW::Twilio::TwiML->new();
    my $resp   = $tw->Response;

    unless( exists $list{$choice} ) {
        $resp->Say({voice => 'woman'},
                   "Sorry, that's not a valid option.");

        $resp->Redirect({method => 'GET'}, "/menu");

        $self->render(format => 'xml',
                      text   => $tw->to_string);
        return;
    }

Twilio always passes a Digits parameter to URLs it fetches as a result of a Gather action (see Twilio's Gather verb documentation); Digits contains the digit or digits pressed by the caller. HTTP GET (and POST) parameters are available to Mojolicious through the param() method.

The code then checks to see if the option the caller has selected exists. If it doesn't, it generates a TwiML response to tell Twilio to Say to the caller "Sorry, that's not a valid option." then sends a redirect back to the main menu.

It's polite to let the caller know what's going on, so add a status message:

$resp->Say({voice => 'woman'},
           "I'll try connecting you now.");

Remember that all this code does is build the TwiML response object; only when the complete, stringified TwiML object is sent to Twilio's application server does the document have any effect on the application flow.

The next step is to look up the caller's choice in the %list hash. If the caller's selection had a phone number associated with it, create a Dial, Play, or Say TwiML object, depending on whether %list specifies a number to call, a URL to an audio file to fetch and play, or neither (respectively):

    if( $list{$choice}->{number} ) {
        $resp->Dial($list{$choice}->{number});
    }

    elsif( $list{$choice}->{recording} ) {
        $resp->Play($list{$choice}->{recording});
    }

    else {
        $resp->Say({voice => 'woman'},
                   "Sorry, that option isn't working.");
        $resp->Redirect({method => 'GET'}, "/menu");
    }

Finally, invoke Mojolicious's render() method to create an XML Content-type header and send the stringified TwiML object to stdout (which Twilio's application server will read):

    $self->render(format => 'xml',
                  text   => $tw->to_string);
};

Here is the full source for your enjoyment. If you've already set Twilio's Voice URL in uour Sandbox, nothing further needs to be done on Twilio's side. Just upload the new application to the web server and start it (Mojolicious supports a variety of deployment options including Morbo, CGI, FastCGI, or Plack).

You've made a simple phone menu here, but you've only learned a few of Twilio's TwiML verbs. With WWW::Twilio::API, WWW::Twilio::TwiML, and Mojolicious you can also create conference rooms, make voice recordings, send and receive SMS messages, reject calls from unwanted numbers, and do other useful actions in just a few lines of code.

Perl can make your phone ring.

Twilio allows developers to write applications that can make and receive voice calls or SMS messages (though Twilio can do many other interesting telephony things). Twilio’s RESTful API, text-to-speech synthesizer, speech transcription services, and Javascript client make it easy to knock out a conference call application, an in-browser customer service voice application, a weather-by-SMS application, reminder by phone—anything, really—in minutes. This article shows how to make a couple of small applications, one to help you pronounce words correctly and the other to transcribe awkward condiment phone survey answers.

Twilio Setup

First, head over to Twilio.com and click the “Try Twilio Free” link. While inbound calls cost US $0.01 per minute and outbound calls cost US $0.02 per minute, Twilio has historically given new users a generous account balance to start with for free (currently US $30)—it’s plenty of credit to kick the tires and take it for a spin.

Go ahead and register (I’ll wait here). When you’ve finished, you’ll have an account SID (beginning with “AC”) and an auth token, available from your Twilio Dashboard. These are your Twilio API username and password; you’ll need them for any API application you write.

Twilio Basics

The Twilio website is full of well-organized documentation and sample applications. I recommend starting with “How It Works” (one of the main navigation links on the home page). Browse the documentation under “Docs” as well.

H. H. Munroe said, “A little inaccuracy sometimes saves tons of explanation.” Keeping that in mind, inbound calls (calls to a Twilio number) work like this:


Inbound calls to a Twilio number

1) the user calls “555-867-5309” on their phone

2) Twilio accepts the call, then makes an HTTP POST to http://example.com/jenny.xml

3) example.com responds with a “TwiML” document (TwiML is a simple XML language that describes how Twilio will interact with callers):

<?xml version="1.0" encoding="UTF-8" ?>
<Response>
  <Say voice="woman">This is Jenny!</Say>
</Response>

4) Twilio’s TwiML parser and text-to-speech synthesizer read this document and then says to the user (in a voice from the uncanny valley), “This is Jenny!”

When you setup a sandbox number, you tell Twilio to map a URL to that number. Twilio will GET/POST to that URL when receiving calls.

Outbound calls (calls from a Twilio number) work like this:


Outbound calls from a Twilio number

1) An application makes an HTTP POST to Twilio’s “Calls” API with the parameters:

From=+15558675309
To=+19991234567
Url=http://example.com/jenny.xml

2) Twilio places the call and waits for an answer

3) Once the user answers, Twilio retrieves the URL specified in the POST (which should return a TwiML document)

4) Twilio parses the TwiML document and passes it to the text-to-speech synthesizer

5) The synthesizer says to the user “This is Jenny!”

Twilio can also record voice input, transcribe it, send and receive SMS messages, make conference calls, and a few other useful things, all using the same familiar RESTful API and TwiML.

Twilio, meet Perl

CPAN makes writing Twilio applications easy, thanks to WWW::Twilio::API. My (current) favorite way to install CPAN modules comes from the Mojolicious project:

curl -L cpanmin.us | perl - WWW::Twilio::API

cpanmin.us returns a Perl program which handles all of the build dependencies for you. If you’re leery of running code from a website directly on the command line, install App::cpanminus and use its cpanm program instead.

If you’re like me and don’t want to mess up your clean development environment, tell cpanmin.us or cpanm to install things into a temporary location:

$ mkdir ~/perl-test
$ export PERL5LIB=~/perl-test/lib/perl5
$ curl -L cpanmin.us | perl - --local-lib=~/perl-test WWW::Twilio::API
# or
$ cpanm --local-lib=~/perl-test WWW::Twilio::API

See How to install CPAN modules for more information and options.

Getting all the necessary Twilio information

After you install WWW::Twilio::API, but before you make your first call, you need several pieces of information from your Twilio Dashboard:

  • AccountSid: this is a long string begining with “AC”

  • AuthToken: another long hex string, next to the AccountSid; you may have to click a lock icon to reveal it

  • Sandbox number: found on the bottom half of the Dashboard page under “Sandbox App”

That’s all! Now you’re ready to go.

Your first phone call

Fire up your favorite editor:

#!/usr/bin/env perl

use strict;
use warnings;
use WWW::Twilio::API;

my $twilio = WWW::Twilio::API->( AccountSid  => 'ACxxxxxxxxx',
                                 AuthToken   => 'xxxxxxxxxxx',
                                 API_VERSION => '2010-04-01' );

## A hollow voice says 'plugh'
my $response = $twilio->POST( 'Calls',
                              To   => '+15556667777', ## maybe your cell phone
                              From => '+12223334444', ## your Twilio sandbox
                              Url  => 'http://twimlets.com/message?'
                                    . 'Message%5B0%5D=plugh' );

print STDERR $response->{content};

That’s the entire application. Run it, and if all went well, you should see a long XML string returned which resembles:

<?xml version="1.0"?>
<TwilioResponse>
  <Call>
    <To>+15556667777</To>
    <From>+12223334444</From>
    <Status>queued</Status>
    <Direction>outbound-api</Direction>
  </Call>
</TwilioResponse>

… and then your phone should ring. I’ve always wondered how to pronounce “plugh”—now I know.

What could possibly go wrong?

Early on in Twilio development, you’re likely to experience a few little gotchas. For example, you might get this message:

LWP will support https URLs if the LWP::Protocol::https module
is installed.

That’s LWP telling you to install LWP::Protocol::https. (It will also install or update a few other modules, including Net::SSLeay).

If you see XML after running the script, your development environment is probably fine. You might instead see:

<TwilioResponse>
  <RestException>
    <Status>401</Status>
    <Message>Authenticate</Message>
    <Code>20003</Code>
    <MoreInfo>http://www.twilio.com/docs/errors/20003</MoreInfo>
  </RestException>
</TwilioResponse>

Notice the “401”? If you’re familiar with HTTP status codes, you might remember that 401 means “Unauthorized”. You either didn’t present any authorization information, or it was incorrect. In this case, it usually means the AccountSid or AuthToken are incorrect. Log into Twilio.com, go to the Dashboard and make sure your AccountSid and AuthToken are correct.

What else can we do?

Everything depends on what you put in for the Url parameter. You can browse some of the free applications at Twilio Labs, though most of those are for inbound calls.

Here’s a silly example of using the voicemail Twimlet to conduct a brief phone survey and have the callee’s response transcribed and emailed. Start by changing the Url line in the POST:

my $email    = 'you@example.com';  ## your email
my $msg      = 'Please+tell+us+what+you+think+of+Tabasco+sauce';
my $response = $twilio->POST( 'Calls',
                              To   => '+15556667777',
                              From => '+12223334444',
                              Url  => "http://twimlets.com/voicemail?"
                                    . "Email=$email&Message=$msg" );

Note that this Twimlet’s arguments are case-sensitive. ‘Email’ and ‘Message’ are not the same as ‘email’ and ‘message’. Make sure you use the correct case.

Also, be sure to substitute your phone number for the To parameter and your Twilio Sandbox phone number (also found on your Twilio Dashboard) for the From parameter. Twilio phone numbers always use the international calling prefix (e.g., United States numbers use “+1” followed by the three digit area code followed by the seven digit phone number).

When you run this, you’ll get a call from Twilio asking you to share your insights into Tabasco sauce. Please be honest. Once you’ve given your opinion, Twilio will then transcribe your message and email it to the email address you specified.

I’m sending out an SMS

SMS messages are even easier: no TwiML needed. Instead of the Calls API, use the SMS/Messages API:

my $response = $twilio->POST( 'SMS/Messages',
                              To   => '+15556667777',
                              From => '+12223334444',
                              Body => 'Rescue me before '
                                   .  'I fall into despair' );

That’s all you have to do to send an SMS message using Twilio (though the 160 character limit applies).

Conclusion

Twilio and Perl make a potent pair: so much is possible with so little code. The next installment will cover writing larger applications with TwiML.

By now, you may have read Considerations on Using Unicode Properly in Modern Perl Applications. Still think doing things correctly is easy? Tom Christiansen demonstrates that even sorting can be trickier than you think.

NOTE: The following is an excerpt from the draft manuscript of Programming Perl, 4ᵗʰ edition

Calling sort without a comparison function is quite often the wrong thing to do, even on plain text. That's because if you use a bare sort, you can get really strange results. It's not just Perl either: almost all programming languages work this way, even the shell command. You might be surprised to find that with this sort of nonsense sort, ‹B› comes before ‹a› not after it, ‹é› comes before ‹d›, and ‹ff› comes after ‹zz›. There's no end to such silliness, either; see the default sort tables at the end of this article to see what I mean.

There are situations when a bare sort is appropriate, but fewer than you think. One scenario is when every string you're sorting contains nothing but the 26 lowercase (or uppercase, but not both) Latin letters from ‹a-z›, without any whitespace or punctuation.

Another occasion when a simple, unadorned sort is appropriate is when you have no other goal but to iterate in an order that is merely repeatable, even if that order should happen to be completely arbitrary. In other words, yes, it's garbage, but it's the same garbage this time as it was last time. That's because the default sort resorts to an unmediated cmp operator, which has the "predictable garbage" characteristics I just mentioned.

The last situation is much less frequent than the first two. It requires that the things you're sorting be special‐purpose, dedicated binary keys whose bit sequences have with excruciating care been arranged to sort in some prescribed fashion. This is also the strategy for any reasonable use of the cmp operator.

So what's wrong with sort anyway?

I know, I know. I can hear everyone saying, "But it's called sort, so how could that ever be wrong?" Sure it's called sort, but you still have to know how to use it to get useful results out. Probably the most surprising thing about sort is that it does not by default do an alphabetic, an alphanumeric, or a numeric sort. What it actually does is something else altogether, and that something else is of surprisingly limited usefulness.

Imagine you have an array of records. It does you virtually no good to write:

@sorted_recs = sort @recs;

Because Perl's cmp operator does only a bit comparison not an alphabetic one, it does nearly as little good to write your record sort this way:

@srecs = sort {
    $b->{AGE}      <=>  $b->{AGE}
                   ||
    $a->{SURNAME}  cmp  $b->{SURNAME}
} @recs;

The problem is that that cmp for the record's SURNAME field is not an alphabetic comparison. It's merely a code point comparison. That means it works like C's strcmp function or Java's String.compareTo method. Although commonly referred to as a "lexicographic" comparison, this is a gross misnomer: it's about as far away from the way real lexicographers sort dictionary entries as you can get without flipping a coin.

Fortunately, you don't have to come up with your own algorithm for dictionary sorting, because Perl provides a standard class to do this for you: Unicode::Collate. Don't let the name throw you, because while it was first invented for Unicode, it works great on regular ASCII text, too, and does a better job at making lexicographers happy than a plain old sort ever manages.

If you have code that purports to sort text that looks like this:

@sorted_lines = sort @lines;

Then all you have to get a dictionary sort is write instead:

use Unicode::Collate;
@sorted_lines = Unicode::Collate::->new->sort(@lines);

For structured records, like those with ages and surnames in them, you have to be a bit fancier. One way to fix it would be to use the class's own cmp operator instead of the built‐in one.

use Unicode::Collate;
my $collator = Unicode::Collate::->new();
@srecs = sort {
    $b->{AGE}  <=>  $b->{AGE}
          ||
    $collator->cmp( $a->{SURNAME}, $b->{SURNAME} )
} @recs;

However, that makes a fairly expensive method call for every possible comparison. Because Perl's adaptive merge sort algorithm usually runs in O(n · log n) time given n items, and because each comparison requires two different computed keys, that can be a lot of duplicate effort. Our sorting class therefore provide a convenient getSortKey method that calculates a special binary key which you can cache and later pass to the normal cmp operator on your own. This trick lets you use cmp yet get a truly alphabetic sort out of it for a change.

Here is a simple but sufficient example of how to do that:

use Unicode::Collate;
my $collator = Unicode::Collate::->new();

# first calculate the magic sort key for each text field, and cache it
for my $rec (@recs) {
    $rec->{SURNAME_key} = $collator->getSortKey( $rec->{SURNAME} );
} 

# now sort the records as before, but for the surname field,
# use the cached sort key instead
@srecs = sort {
    $b->{AGE}          <=>  $b->{AGE}
                      ||
    $a->{SURNAME_key}  cmp  $b->{SURNAME_key}
} @recs;

That's what I meant about very carefully preparing a mediated sort key that contains the precomputed binary key.

English Card Catalogue Sorts

The simple code just demonstrated assumes you want to sort names the same way you do regular text. That isn't a good assumption, however. Many countries, languages, institutions, and sometimes even librarians have their own notions about how a card catalogue or a phonebook ought to be sorted.

For example, in the English language, surnames with Scottish patronymics starting with ‹Mc› or ‹Mac›, like MacKinley and McKinley, not only count as completely identical synonyms for sorting purposes, they go before any other surname that begins with ‹M›, and so precede surnames like Mables or Machado.

Yes, really.

That means that the following names are sorted correctly -- for English:

Lewis, C.S.
McKinley, Bill
MacKinley, Ron
Mables, Martha
Machado, José
Macon, Bacon

Yes, it's true. Check out your local large English‐language bookseller or library -- presuming you can find one. If you do, best make sure to blow the dust off first.

Sorting Spanish Names

It's a good thing those names follow English rules for sorting names. If this were Spanish, we would have to deal with double‐barrelled surnames, where the patronym sorts before the matronym, which in turn sorts before any given names. That means that if Señor Machado's full name were, like the poet's, Antonio Cipriano José María y Francisco de Santa Ana Machado y Ruiz, then you would have to sort him with the other Machados but then consider Ruiz before Antonio if there were any other Machados. Similarly, the poet Federico del Sagrado Corazón de Jesús García Lorca sorts before the writer Gabriel José de la Concordia García Márquez.

On the other hand, if your records are not full multifield hashes but only simple text that don't happen to be surnames, your task is a lot simpler, since now all you have to is get the cmp operator to behave sensibly. That you can do easily enough this way:

use Unicode::Collate;
@sorted_text = Unicode::Collate::->new->sort(@text);

Sorting Text, Not Binary

Imagine you had this list of German‐language authors:

@germans = qw{
    Böll
    Born
    Böhme
    Bodmer
    Brandis
    Böttcher
    Borchert
    Bobrowski
};

If you just sorted them with an unmediated sort operator, you would get this utter nonsense:

Bobrowski
Bodmer
Borchert
Born
Brandis
Brant
Böhme
Böll
Böttcher

Or maybe this equally nonsensical answer:

Bobrowski
Bodmer
Borchert
Born
Böll
Brandis
Brant
Böhme
Böttcher

Or even this still completely nonsensical answer:

Bobrowski
Bodmer
Borchert
Born
Böhme
Böll
Brandis
Brant
Böttcher

The crucial point to all that is that it's text not binary, so not only can you never judge what its bit patterns hold just by eyeballing it, more importantly, it has special rules to make it sort alphabetically (some might say sanely), an ordering no naïve code‐point sort will never come even close to getting right, especially on Unicode.

The correct ordering is:

Bobrowski
Bodmer
Böhme
Böll
Borchert
Born
Böttcher
Brandis
Brant

And that is precisely what

use Unicode::Collate;
@sorted_germans = Unicode::Collate::->new->sort(@german_names);

gives you: a correctly sorted list of those Germans' names.

Sorting German Names

Hold on, though.

Correct in what language? In English, yes, the order given is now correct. But considering that these authors wrote in the German language, it is quite conceivable that you should be following the rules for ordering German names in German, not in English. That produces this ordering:

Bobrowski
Bodmer
Böhme
Böll
Böttcher
Borchert
Born
Brandis
Brant

How come Böttcher now came before Borchert? Because Böttcher is supposed to be the same as Boettcher. In a German phonebook or other German list of German names, things like ‹ö› and ‹oe› are considered synonyms, which is not at all how it works in English. To get the German phonebook sort, you merely have to modify your constructor this way:

use Unicode::Collate::Locale;
@sorted_germans = Unicode::Collate::Locale::
                      ->new(locale => "de_phonebook")
                      ->sort(@german_names);

Isn't this fun?

Be glad you're not sorting names. Sorting names is hard.

Default Sort Tables

Here are most of the Latin letters, ordered using the default sort:

A B C D E F G H I J K L M N O P Q R S T U V W X Y Z a b c d e f g h i j 
k l m n o p q r s t u v w x y z ª º À Á Â Ã Ä Å Æ Ç È É Ê Ë Ì Í Î Ï Ð Ñ 
ҠӠԠՠ֠ؠ٠ڠ۠ܠݠޠߠà á â ã ä å æ ç è é ê ë ì í î ï ð ñ ò ó ô õ ö 
ø ù ú û ü ý þ ÿ Ā ā Ă ă Ą ą Ć ć Ĉ ĉ Ċ ċ Č č Ď ď Đ đ Ē ē Ĕ ĕ Ė ė Ę ę Ě ě 
Ĝ ĝ Ğ ğ Ġ ġ Ģ ģ Ĥ ĥ Ħ ħ Ĩ ĩ Ī ī Ĭ ĭ Į į İ ı IJ ij Ĵ ĵ Ķ ķ ĸ Ĺ ĺ Ļ ļ Ľ ľ Ŀ 
ŀ Ł ł Ń ń Ņ ņ Ň ň Ŋ ŋ Ō ō Ŏ ŏ Ő ő Œ œ Ŕ ŕ Ŗ ŗ Ř ř Ś ś Ŝ ŝ Ş ş Š š Ţ ţ Ť 
ť Ŧ ŧ Ũ ũ Ū ū Ŭ ŭ Ů ů Ű ű Ų ų Ŵ ŵ Ŷ ŷ Ÿ Ź ź Ż ż Ž ž ſ ƀ Ɓ Ƃ ƃ Ƈ ƈ Ɖ Ɗ Ƌ 
ƌ ƍ Ǝ Ə Ɛ Ƒ ƒ Ɠ Ɣ ƕ Ɩ Ɨ Ƙ ƙ ƚ ƛ Ɯ Ɲ ƞ Ƥ ƥ Ʀ ƫ Ƭ ƭ Ʈ Ư ư Ʊ Ʋ Ƴ ƴ Ƶ ƶ Ʒ Ƹ 
ƹ ƺ ƾ ƿ DŽ Dž dž LJ Lj lj NJ Nj nj Ǎ ǎ Ǐ ǐ Ǒ ǒ Ǔ ǔ Ǖ ǖ Ǘ ǘ Ǚ ǚ Ǜ ǜ ǝ Ǟ ǟ Ǡ ǡ Ǣ ǣ 
Ǥ ǥ Ǧ ǧ Ǩ ǩ Ǫ ǫ Ǭ ǭ Ǯ ǯ ǰ DZ Dz dz Ǵ ǵ Ƿ Ǹ ǹ Ǻ ǻ Ǽ ǽ Ǿ ǿ Ȁ ȁ Ȃ ȃ Ȅ ȅ Ȇ ȇ Ȉ 
ȉ Ȋ ȋ Ȍ ȍ Ȏ ȏ Ȑ ȑ Ȓ ȓ Ȕ ȕ Ȗ ȗ Ș ș Ț ț Ȝ ȝ Ȟ ȟ Ƞ ȡ Ȥ ȥ Ȧ ȧ Ȩ ȩ Ȫ ȫ Ȭ ȭ Ȯ 
ȯ Ȱ ȱ Ȳ ȳ ȴ ȵ ȶ ȷ Ⱥ Ȼ ȼ Ƚ Ⱦ ɐ ɑ ɒ ɓ ɕ ɖ ɗ ɘ ə ɚ ɛ ɜ ɝ ɞ ɟ ɠ ɡ ɢ ɣ ɤ ɥ ɦ 
ɧ ɨ ɩ ɪ ɫ ɬ ɭ ɮ ɯ ɰ ɱ ɲ ɳ ɴ ɶ ɹ ɺ ɻ ɼ ɽ ɾ ɿ ʀ ʁ ʂ ʃ ʄ ʅ ʆ ʇ ʈ ʉ ʊ ʋ ʌ ʍ 
ʎ ʏ ʐ ʑ ʒ ʓ ʙ ʚ ʛ ʜ ʝ ʞ ʟ ʠ ʣ ʤ ʥ ʦ ʧ ʨ ʩ ʪ ʫ ˡ ˢ ˣ ᴀ ᴁ ᴂ ᴃ ᴄ ᴅ ᴆ ᴇ ᴈ ᴉ 
ᴊ ᴋ ᴌ ᴍ ᴎ ᴏ ᴑ ᴓ ᴔ ᴘ ᴙ ᴚ ᴛ ᴜ ᴝ ᴞ ᴟ ᴠ ᴡ ᴢ ᴣ ᴬ ᴭ ᴮ ᴯ ᴰ ᴱ ᴲ ᴳ ᴴ ᴵ ᴶ ᴷ ᴸ ᴹ ᴺ 
ᴻ ᴼ ᴾ ᴿ ᵀ ᵁ ᵂ ᵃ ᵄ ᵅ ᵆ ᵇ ᵈ ᵉ ᵊ ᵋ ᵌ ᵍ ᵎ ᵏ ᵐ ᵑ ᵒ ᵖ ᵗ ᵘ ᵙ ᵚ ᵛ ᵢ ᵣ ᵤ ᵥ ᵫ ᵬ ᵭ 
ᵮ ᵯ ᵰ ᵱ ᵲ ᵳ ᵴ ᵵ ᵶ Ḁ ḁ Ḃ ḃ Ḅ ḅ Ḇ ḇ Ḉ ḉ Ḋ ḋ Ḍ ḍ Ḏ ḏ Ḑ ḑ Ḓ ḓ Ḕ ḕ Ḗ ḗ Ḙ ḙ Ḛ 
ḛ Ḝ ḝ Ḟ ḟ Ḡ ḡ Ḣ ḣ Ḥ ḥ Ḧ ḧ Ḩ ḩ Ḫ ḫ Ḭ ḭ Ḯ ḯ Ḱ ḱ Ḳ ḳ Ḵ ḵ Ḷ ḷ Ḹ ḹ Ḻ ḻ Ḽ ḽ Ḿ 
ḿ Ṁ ṁ Ṃ ṃ Ṅ ṅ Ṇ ṇ Ṉ ṉ Ṋ ṋ Ṍ ṍ Ṏ ṏ Ṑ ṑ Ṓ ṓ Ṕ ṕ Ṗ ṗ Ṙ ṙ Ṛ ṛ Ṝ ṝ Ṟ ṟ Ṡ ṡ Ṣ 
ṣ Ṥ ṥ Ṧ ṧ Ṩ ṩ Ṫ ṫ Ṭ ṭ Ṯ ṯ Ṱ ṱ Ṳ ṳ Ṵ ṵ Ṷ ṷ Ṹ ṹ Ṻ ṻ Ṽ ṽ Ṿ ṿ Ẁ ẁ Ẃ ẃ Ẅ ẅ Ẇ 
ẇ Ẉ ẉ Ẋ ẋ Ẍ ẍ Ẏ ẏ Ẑ ẑ Ẓ ẓ Ẕ ẕ ẖ ẗ ẘ ẙ ẚ ẛ ẞ ẟ Ạ ạ Ả ả Ấ ấ Ầ ầ Ẩ ẩ Ẫ ẫ Ậ 
ậ Ắ ắ Ằ ằ Ẳ ẳ Ẵ ẵ Ặ ặ Ẹ ẹ Ẻ ẻ Ẽ ẽ Ế ế Ề ề Ể ể Ễ ễ Ệ ệ Ỉ ỉ Ị ị Ọ ọ Ỏ ỏ Ố 
ố Ồ ồ Ổ ổ Ỗ ỗ Ộ ộ Ớ ớ Ờ ờ Ở ở Ỡ ỡ Ợ ợ Ụ ụ Ủ ủ Ứ ứ Ừ ừ Ử ử Ữ ữ Ự ự Ỳ ỳ Ỵ 
ỵ Ỷ ỷ Ỹ ỹ K Å Ⅎ ⅎ Ⅰ Ⅱ Ⅲ Ⅳ Ⅴ Ⅵ Ⅶ Ⅷ Ⅸ Ⅹ Ⅺ Ⅻ Ⅼ Ⅽ Ⅾ Ⅿ ⅰ ⅱ ⅲ ⅳ ⅴ 
ⅵ ⅶ ⅷ ⅸ ⅹ ⅺ ⅻ ⅼ ⅽ ⅾ ⅿ ff fi fl ffi ffl ſt st A B C D E F G H I
J K L M N O P Q R S T U V W X Y Z a b c d e f g h i
j k l m n o p q r s t u v w x y z

As you can see, those letters are scattered all over the place. Sure, it's not completely random, but it's not useful either, because it is full of arbitrary placement that makes no alphabetical sense. That's because it is not an alphabetic sort at all. However, with the special kind of sort I've just shown you above, the ones that call the sort method from the Unicode::Collate class, you do get an alphabetic sort. Using that method, the Latin letters I just showed you now come out in alphabetical order, which is like this:

a a A A ª ᵃ ᴬ á Á à À ă Ă ắ Ắ ằ Ằ ẵ Ẵ ẳ Ẳ â Â ấ Ấ ầ Ầ ẫ Ẫ ẩ Ẩ ǎ Ǎ å Å 
Å ǻ Ǻ ä Ä ǟ Ǟ ã Ã ȧ Ȧ ǡ Ǡ ą Ą ā Ā ả Ả ȁ Ȁ ȃ Ȃ ạ Ạ ặ Ặ ậ Ậ ḁ Ḁ æ Æ ᴭ ǽ Ǽ 
ǣ Ǣ ẚ ᴀ Ⱥ ᴁ ᴂ ᵆ ɐ ᵄ ɑ ᵅ ɒ b b B B ᵇ ᴮ ḃ Ḃ ḅ Ḅ ḇ Ḇ ʙ ƀ ᴯ ᴃ ᵬ ɓ Ɓ ƃ Ƃ c 
c ⅽ C C Ⅽ ć Ć ĉ Ĉ č Č ċ Ċ ç Ç ḉ Ḉ ᴄ ȼ Ȼ ƈ Ƈ ɕ d d ⅾ D D Ⅾ ᵈ ᴰ ď Ď ḋ 
Ḋ ḑ Ḑ ḍ Ḍ ḓ Ḓ ḏ Ḏ đ Đ ð Ð dz ʣ Dz DZ dž Dž DŽ ʥ ʤ ᴅ ᴆ ᵭ ɖ Ɖ ɗ Ɗ ƌ Ƌ ȡ ẟ e e E 
E ᵉ ᴱ é É è È ĕ Ĕ ê Ê ế Ế ề Ề ễ Ễ ể Ể ě Ě ë Ë ẽ Ẽ ė Ė ȩ Ȩ ḝ Ḝ ę Ę ē Ē ḗ 
Ḗ ḕ Ḕ ẻ Ẻ ȅ Ȅ ȇ Ȇ ẹ Ẹ ệ Ệ ḙ Ḙ ḛ Ḛ ᴇ ǝ Ǝ ᴲ ə Ə ᵊ ɛ Ɛ ᵋ ɘ ɚ ɜ ᴈ ᵌ ɝ ɞ ʚ ɤ 
f f F F ḟ Ḟ ff ffi ffl fi fl ʩ ᵮ ƒ Ƒ ⅎ Ⅎ g g G G ᵍ ᴳ ǵ Ǵ ğ Ğ ĝ Ĝ ǧ Ǧ ġ Ġ ģ 
Ģ ḡ Ḡ ɡ ɢ ǥ Ǥ ɠ Ɠ ʛ ɣ Ɣ h h H H ᴴ ĥ Ĥ ȟ Ȟ ḧ Ḧ ḣ Ḣ ḩ Ḩ ḥ Ḥ ḫ Ḫ ẖ ħ Ħ ʜ 
ƕ ɦ ɧ i i ⅰ I I Ⅰ ᵢ ᴵ í Í ì Ì ĭ Ĭ î Î ǐ Ǐ ï Ï ḯ Ḯ ĩ Ĩ İ į Į ī Ī ỉ Ỉ ȉ 
Ȉ ȋ Ȋ ị Ị ḭ Ḭ ⅱ Ⅱ ⅲ Ⅲ ij IJ ⅳ Ⅳ ⅸ Ⅸ ı ɪ ᴉ ᵎ ɨ Ɨ ɩ Ɩ j j J J ᴶ ĵ Ĵ ǰ ȷ ᴊ 
ʝ ɟ ʄ k k K K K ᵏ ᴷ ḱ Ḱ ǩ Ǩ ķ Ķ ḳ Ḳ ḵ Ḵ ᴋ ƙ Ƙ ʞ l l ⅼ L L Ⅼ ˡ ᴸ ĺ Ĺ 
ľ Ľ ļ Ļ ḷ Ḷ ḹ Ḹ ḽ Ḽ ḻ Ḻ ł Ł ŀ Ŀ lj Lj LJ ʪ ʫ ʟ ᴌ ƚ Ƚ ɫ ɬ ɭ ȴ ɮ ƛ ʎ m m ⅿ M 
M Ⅿ ᵐ ᴹ ḿ Ḿ ṁ Ṁ ṃ Ṃ ᴍ ᵯ ɱ n n N N ᴺ ń Ń ǹ Ǹ ň Ň ñ Ñ ṅ Ṅ ņ Ņ ṇ Ṇ ṋ Ṋ ṉ 
Ṉ nj Nj NJ ɴ ᴻ ᴎ ᵰ ɲ Ɲ ƞ Ƞ ɳ ȵ ŋ Ŋ ᵑ o o O O º ᵒ ᴼ ó Ó ò Ò ŏ Ŏ ô Ô ố Ố ồ 
Ồ ỗ Ỗ ổ Ổ ǒ Ǒ ö Ö ȫ Ȫ ő Ő õ Õ ṍ Ṍ ṏ Ṏ ȭ Ȭ ȯ Ȯ ȱ Ȱ ø Ø ǿ Ǿ ǫ Ǫ ǭ Ǭ ō Ō ṓ 
Ṓ ṑ Ṑ ỏ Ỏ ȍ Ȍ ȏ Ȏ ớ Ớ ờ Ờ ỡ Ỡ ở Ở ợ Ợ ọ Ọ ộ Ộ œ Œ ᴏ ᴑ ɶ ᴔ ᴓ p p P P ᵖ 
ᴾ ṕ Ṕ ṗ Ṗ ᴘ ᵱ ƥ Ƥ q q Q Q ʠ ĸ r r R R ᵣ ᴿ ŕ Ŕ ř Ř ṙ Ṙ ŗ Ŗ ȑ Ȑ ȓ Ȓ ṛ 
Ṛ ṝ Ṝ ṟ Ṟ ʀ Ʀ ᴙ ᵲ ɹ ᴚ ɺ ɻ ɼ ɽ ɾ ᵳ ɿ ʁ s s S S ˢ ś Ś ṥ Ṥ ŝ Ŝ š Š ṧ Ṧ ṡ 
Ṡ ş Ş ṣ Ṣ ṩ Ṩ ș Ș ſ ẛ ß ẞ st ſt ᵴ ʂ ʃ ʅ ʆ t t T T ᵗ ᵀ ť Ť ẗ ṫ Ṫ ţ Ţ ṭ Ṭ 
ț Ț ṱ Ṱ ṯ Ṯ ʨ ƾ ʦ ʧ ᴛ ŧ Ŧ Ⱦ ᵵ ƫ ƭ Ƭ ʈ Ʈ ȶ ʇ u u U U ᵘ ᵤ ᵁ ú Ú ù Ù ŭ Ŭ 
û Û ǔ Ǔ ů Ů ü Ü ǘ Ǘ ǜ Ǜ ǚ Ǚ ǖ Ǖ ű Ű ũ Ũ ṹ Ṹ ų Ų ū Ū ṻ Ṻ ủ Ủ ȕ Ȕ ȗ Ȗ ư Ư 
ứ Ứ ừ Ừ ữ Ữ ử Ử ự Ự ụ Ụ ṳ Ṳ ṷ Ṷ ṵ Ṵ ᴜ ᴝ ᵙ ᴞ ᵫ ʉ ɥ ɯ Ɯ ᵚ ᴟ ɰ ʊ Ʊ v v ⅴ V 
V Ⅴ ᵛ ᵥ ṽ Ṽ ṿ Ṿ ⅵ Ⅵ ⅶ Ⅶ ⅷ Ⅷ ᴠ ʋ Ʋ ʌ w w W W ᵂ ẃ Ẃ ẁ Ẁ ŵ Ŵ ẘ ẅ Ẅ ẇ Ẇ ẉ 
Ẉ ᴡ ʍ x x ⅹ X X Ⅹ ˣ ẍ Ẍ ẋ Ẋ ⅺ Ⅺ ⅻ Ⅻ y y Y Y ý Ý ỳ Ỳ ŷ Ŷ ẙ ÿ Ÿ ỹ Ỹ ẏ 
Ẏ ȳ Ȳ ỷ Ỷ ỵ Ỵ ʏ ƴ Ƴ z z Z Z ź Ź ẑ Ẑ ž Ž ż Ż ẓ Ẓ ẕ Ẕ ƍ ᴢ ƶ Ƶ ᵶ ȥ Ȥ ʐ ʑ 
ʒ Ʒ ǯ Ǯ ᴣ ƹ Ƹ ƺ ʓ ȝ Ȝ þ Þ ƿ Ƿ

Isn't that much nicer?

Romani Ite Domum

In case you're wondering what that last row of distinctly un‐Roman Latin letters might possibly be, they're called respectively ezh ʒ, yogh ȝ, thorn þ, and wynn ƿ. They had to go somewhere, so they ended up getting stuck after ‹z›

Some are still used in certain non‐English (but still Latin) alphabets today, such as Icelandic, and even though you probably won't bump into them in contemporary English texts, you might see some if you're reading the original texts of famous medieval English poems like Beowulf, Sir Gawain and the Green Knight, or Brut.

The last of those, Brut, was written by a fellow named Laȝamon, a name whose third letter is a yogh. Famous though he was, I wouldn't suggest changing your name to ‹Laȝamon› in his honor, as I doubt the phone company would be amused.

Visit the home of the Perl programming language: Perl.org

Sponsored by

Monthly Archives

Powered by Movable Type 5.12