℞ 24: Disabling Unicode-awareness in builtin charclasses

Many regex tutorials gloss over the fact that builtin character classes include far more than ASCII characters. In particular, classes such as "word character" (\w), "word boundary" (\b), "whitespace" (\s), and "digit" (\d) respect Unicode.

Perl 5.14 added the /a regex modifier to disable \w, \b, \s, \d, and the POSIX classes from working correctly on Unicode. This restricts these classes to mach only ASCII characters. Use the re pragma to restrict these claracter classes in a lexical scope:

 use v5.14;
 use re "/a";

... or use the /a modifier to affect a single regex:

 my($num) = $str =~ /(\d+)/a;

You may always use specific un-Unicode properties, such \p{ahex} and \p{POSIX_Digit}. Properties still work normally no matter what charset modifiers (/d /u /l /a /aa) are in effect.

℞ 23: Get character category

Unicode is a set of characters and a list of rules and properties applied to those characters. The Unicode Character Database collects those properties. The core module Unicode::UCD provides access to these properties.

These general properties group characters into groups, such as upper- or lowercase characters, punctuation symbols, math symbols, and more. (See Unicode::UCD's general_categories() for more information.)

The charinfo() function returns a hash reference containing a wealth of information about the Unicode character in question. In particular, its category value contains the short name of a character's category.

To find the general category of a numeric codepoint:

 use Unicode::UCD qw(charinfo);
 my $cat = charinfo(0x3A3)->{category};  # "Lu"

To translate this category into something more human friendly:

 use Unicode::UCD qw( charinfo general_categories );
 my $categories = general_categories();
 my $cat        = charinfo(0x3A3)->{category};  # "Lu"
 my $full_cat   = $categories{ $cat }; # "UppercaseLetter"

℞ 22: Match Unicode linebreak sequence in regex

Unicode defines several characters as providing vertical whitespace, like the carriage return or newline characters. Unicode also gathers several characters under the banner of a linebreak sequence. A Unicode linebreak matches the two-character CRLF grapheme or any of the seven vertical whitespace characters.

As documented in perldoc perlrebackslash, the \R regex backslash sequence matches any Unicode linebreak sequence. (Similarly, the \v sequence matches any single character of vertical whitespace.)

This is useful for dealing with textfiles coming from different operating systems:

 s/\R/\n/g;  # normalize all linebreaks to \n

℞ 21: Unicode case-insensitive comparisons

Unicode is more than an expanded character set. Unicode is a set of rules about how characters behave and a set of properties about each character.

Comparing strings for equivalence often requires normalizing them to a standard form. That normalized form often requires that all characters be in a specific case. ℞ 20: Unicode casing demonstrated that converting between upper- and lower-case Unicode characters is more complicated than simply mapping [A-Z] to [a-z]. (Remember also that many characters have a title case form!)

The proper solution for normalized comparisons is to perform casefolding instead of mapping a subset of some characters to another. Perl 5.16 added a new feature fc(), or "foldcase", to perform Unicode casefolding as the /i pattern modifier has always provided. This feature is available for other Perls thanks to the CPAN module Unicode::CaseFold:

 use feature "fc"; # fc() function is from v5.16
 # OR
 use Unicode::CaseFold;

 # sort case-insensitively
 my @sorted = sort { fc($a) cmp fc($b) } @list;

 # both are true:
 fc("tschüß")  eq fc("TSCHÜSS")
 fc("Σίσυφος") eq fc("ΣΊΣΥΦΟΣ")

Fold cases properly goes into more detail about case folding in Perl.

℞ 20: Unicode casing

Unicode casing is very different from ASCII casing. Some of the complexity of Unicode comes about because Unicode characters may change dramatically when changing from upper to lower case and back. For example, the Greek language has two characters for the lower case sigma, depending on whether the letter is in a medial (σ) or final (ς) position in a word. Greek only has a single upper case sigma (Σ). (Some classical Greek texts from the Hellenistic period use a crescent-shaped variant of the sigma called the lunate sigma, or ϲ.)

Unicode casing is important for changing case and for performing case-insensitive matching:

 uc("henry ⅷ")  # "HENRY Ⅷ"
 uc("tschüß")   # "TSCHÜSS"  notice ß => SS

 # both are true:
 "tschüß"  =~ /TSCHÜSS/i   # notice ß => SS
 "Σίσυφος" =~ /ΣΊΣΥΦΟΣ/i   # notice Σ,σ,ς sameness

℞ 19: Open file with specific encoding

While setting the default Unicode encoding for IO is sensible, sometimes the default encoding is not correct. In this case, specify the encoding for a filehandle manually in the mode option to open or with the binmode operator. Perl's IO layers will handle encoding and decoding for you. This is the normal way to deal with encoded text, not by calling low-level functions.

To specify the encoding of a filehandle opened for input:

    open(my $in_file, "< :encoding(UTF-16)", "wintext");
     # OR
     open(my $in_file, "<", "wintext");
     binmode($in_file, ":encoding(UTF-16)");

     # ...
     my $line = <$in_file>;

To specify the encoding of a filehandle opened for output:

     open($out_file, "> :encoding(cp1252)", "wintext");
     # OR
     open(my $out_file, ">", "wintext");
     binmode($out_file, ":encoding(cp1252)");

     # ...
     print $out_file "some text\n";

More layers than just the encoding can be specified here. For example, the incantation ":raw :encoding(UTF-16LE) :crlf" includes implicit CRLF handling. See PerlIO for more details.

℞ 18: Make all I/O and args default to utf8

The core rule of Unicode handling in Perl is "always encode and decode at the edges of your program".

If you've configured everything such that all incoming and outgoing data uses the UTF-8 encoding, you can make Perl perform the appropriate encoding and decoding for you. As documented in perldoc perlrun, the -C flag and the PERL_UNICODE environment variable are available. Use the S option to make the standard input, output, and error filehandles use UTF-8 encoding. Use the D option to make all other filehandles use UTF-8 encoding. Use the A option to decode @ARGV elements as UTF-8:

     $ perl -CSDA ...
# or
     $ export PERL_UNICODE=SDA

Within your program, you can achieve the same effects with the open pragma to set default encodings on filehandles and the Encode module to decode the elements of @ARGV:

     use open qw(:std :utf8);
     use Encode qw(decode_utf8);
     @ARGV = map { decode_utf8($_, 1) } @ARGV;

℞ 17: Make file I/O default to utf8

If you've ever had the misfortune of seeing the Unicode warning "wide character in print", you may have realized that something forgot to set the appropriate Unicode-capable encoding on a filehandle somewhere in your program. Remember that the rule of Unicode handling in Perl is "always encode and decode at the edges of your program".

You can easily Decode STDIN, STDOUT, and STDERR as UTF-8 by default or Decode STDIN, STDOUT, and STDERR per local settings as a default, or you can use binmode to set the encoding on a specific filehandle.

Alternately, you can set the default encoding on all filehandles through the entire program, or on a lexical basis. As documented in perldoc perlrun, the -C flag and the PERL_UNICODE environment variable are available. Use the D option to make all filehandles default to UTF-8 encoding. That is, files opened without an encoding argument will be in UTF-8:

     $ perl -CD ...
     # or
     $ export PERL_UNICODE=D

The open pragma configures the default encoding of all filehandle operations in its lexical scope:

     use open qw(:utf8);

Note that the open pragma is currently incompatible with the autodie pragma.

℞ 16: Declare STD{IN,OUT,ERR} to be in locale encoding

Always convert to and from your desired encoding at the edges of your programs. This includes the standard filehandles STDIN, STDOUT, and STDERR. While it may be most common for modern operating systems to support UTF-8 in filehandle settings, you may need to use other encodings.

Perl can respect your current locale settings for its default filehandles. Start by installing the Encode::Locale module from the CPAN.

    # cpan -i Encode::Locale
    use Encode;
    use Encode::Locale;

    # or as a stream for binmode or open
    binmode STDIN,  ":encoding(console_in)"  if -t STDIN;
    binmode STDOUT, ":encoding(console_out)" if -t STDOUT;
    binmode STDERR, ":encoding(console_out)" if -t STDERR;

The Encode::Locale module allows you to use "whatever encoding the attached terminal expects" for input and output filehandles attached to terminals. It also allows you to specify "whatever encoding the file system uses for file names"; see the documentation for more.

℞ 15: Declare STD{IN,OUT,ERR} to be UTF-8

Always convert to and from your desired encoding at the edges of your programs. This includes the standard filehandles STDIN, STDOUT, and STDERR.

As documented in perldoc perlrun, the PERL_UNICODE environment variable or the -C command-line flag allow you to tell Perl to encode and decode from and to these filehandles as UTF-8, with the S option:

     $ perl -CS ...
     # or
     $ export PERL_UNICODE=S

Within your program, the open pragma allows you to set the default encoding of these filehandles all at once:

     use open qw(:std :utf8);

Because Perl uses IO layers to implement encoding and decoding, you may also use the binmode operator on filehandles directly:

     binmode(STDIN,  ":utf8");
     binmode(STDOUT, ":utf8");
     binmode(STDERR, ":utf8");

℞ 14: Decode program arguments as locale encoding

While it may be most common in modern operating systems for your command-line arguments to be encoded as UTF-8, @ARGV may use other encodings. If you have configured your system with a proper locale, you may need to decode @ARGV appropriately. Unlike automatic UTF-8 @ARGV decoding, you must do this manually.

Install the Encode::Locale module from the CPAN:

    # cpan -i Encode::Locale
    use Encode qw(locale);
    use Encode::Locale;

    # use "locale" as an arg to encode/decode
    @ARGV = map { decode(locale => $_, 1) } @ARGV;

℞ 13: Decode program arguments as utf8

While the standard Perl Unicode preamble makes Perl's filehandles use UTF-8 encoding by default, filehandles aren't the only sources and sinks of data. The command-line arguments to your programs, available through @ARGV, may also need decoding.

You can have Perl handle this operation for you automatically in two ways, and may do it yourself manually. As documented in perldoc perlrun, the -C flag controls Unicode features. Use the A modifier for Perl to treat your arguments as UTF-8 strings:

     $ perl -CA ...

You may, of course, use -C on the shebang line of your programs.

The second approach is to use the PERL_UNICODE environment variable. It takes the same values as the -C flag; to get the same effect as -CA, write:

     $ export PERL_UNICODE=A

You may temporarily disable this automatic Unicode treatment with PERL_UNICODE=0.

Finally, you may decode the contents of @ARGV yourself manually with the Encode module:

    use Encode qw(decode_utf8);
    @ARGV = map { decode_utf8($_, 1) } @ARGV;

℞ 12: Explicit encode/decode

While the standard Perl Unicode preamble makes Perl's filehandles use UTF-8 encoding by default, filehandles aren't the only sources and sinks of data. On rare occasions, such as a database read, you may be given encoded text you need to decode.

The core Encode module offers two functions to handle these conversions. (Remember that decode() means to convert octets from a known encoding into Perl's internal Unicode form and encode() means to convet from Perl's internal form into a known encoding.)

  use Encode qw(encode decode);

  # given $bytes, containing octets in a known encoding
  my $chars = decode("shiftjis", $bytes, 1);

  # given $chars, a string encoded in Perl's internal format
  my $bytes = encode("MIME-Header-ISO_2022_JP", $chars, 1);

For streams all in the same encoding, don't use encode/decode; instead set the file encoding when you open the file or immediately after with binmode as described in a future reference. Remember the canonical rule of Unicode: always encode/decode at the edges of your application.

℞ 11: Names of CJK codepoints

CJK refers to Chinese, Japanese, and Korean. In the context of Unicode, it usually refers to the Han ideographs used in the modern Chinese and Japanese writing systems. As you can expect, pictoral languages such as Chinese make Unicode handling more complex.

Sinograms like "東京" come back with character names of CJK UNIFIED IDEOGRAPH-6771 and CJK UNIFIED IDEOGRAPH-4EAC, because their "names" vary between languages. The CPAN Unicode::Unihan module has a large database for decoding these (and a whole lot more), provided you know how to understand its output.

 # cpan -i Unicode::Unihan
 use Unicode::Unihan;
 my $str   = "東京";
 my $unhan = Unicode::Unihan->new;
 for my $lang (qw(Mandarin Cantonese Korean JapaneseOn JapaneseKun)) {
     printf "CJK $str in %-12s is ", $lang;
     say $unhan->$lang($str);
 }

prints:

 CJK 東京 in Mandarin     is DONG1JING1
 CJK 東京 in Cantonese    is dung1ging1
 CJK 東京 in Korean       is TONGKYENG
 CJK 東京 in JapaneseOn   is TOUKYOU KEI KIN
 CJK 東京 in JapaneseKun  is HIGASHI AZUMAMIYAKO

If you have a specific romanization scheme in mind, use the specific module:

 # cpan -i Lingua::JA::Romanize::Japanese
 use Lingua::JA::Romanize::Japanese;
 my $k2r = Lingua::JA::Romanize::Japanese->new;
 my $str = "東京";
 say "Japanese for $str is ", $k2r->chars($str);

prints:

 Japanese for 東京 is toukyou

℞ 10: Custom named characters

As several other recipes demonstrate, the charnames pragma offers considerable power to use and manipulate Unicode characters by their names. Its :alias option allows you to give your own lexically scoped nicknames to existing characters, or even to give unnamed private-use characters useful names:

 use charnames ":full", ":alias" => {
     ecute => "LATIN SMALL LETTER E WITH ACUTE",
     "APPLE LOGO" => 0xF8FF, # private use character
 };

 "\N{ecute}"
 "\N{APPLE LOGO}"

You may even override existing names (lexically, of course) with different characters.

This feature has some limitations. For best effect, aliases should hew to the rules of ASCII identifiers and must not resemble regex quantifiers. You can only alias one character at a time; other options exist to give a character sequence an alias.

As always, the documentation of the charnames pragma offers more details.