November 2004 Archives

Perl Debugger Quick Reference

Perl's debugger is powerful but somewhat esoteric. The core perldebtut tutorial is a good introduction, but who wants to skim through pages of documentation when you only need a quick refresher on the syntax of a command?

Let this printable (PDF) Perl Debugger Quick Reference card, excerpted from Richard Foley's Perl Debugger Pocket Reference, be your guide instead.

Editor's note: Unix users, we've had reports that Adobe Acrobat 5 has trouble opening this file. We're looking into this. In the meantime, xpdf and other tools appear to work correctly.

This Week in Perl 6, November 9 - 15 2004

All~

Welcome to yet another Monday summary. This would have been a Sunday summary, but Avernum (from Spiderweb Software) forcibly prevented it. As usual, we will start out with Perl 6 Language.

Perl 6 Language

Modules and Exports

Aaron Sherman wanted to know some specifics about exporting things from modules. In particular, he did not want to have to retype a whole bunch of things. Larry gave an answer indicating that some of what Aaron wants should occur, but some things are difficult for good reason.

AUTOCLASS(DEF)? Hooks

Rod Adams wanted to know if he could get AUTOCLASS and AUTOCLASSDEF hooks. Larry told him that he probably could. Rod also wanted module name aliasing. Larry told him that he could. Rod wanted to have module names with arbitrary strings. Larry told him that he could. Rod wanted to be able to load multiple versions of the same module. Larry told him that he could. The summarizer wanted each question of a different focus to be in a different message. Hopefully, Larry will tell him that they should...

Matching the nth Occurrences

Sudarshan Gaikaiwari wanted to know how to match the "second last" occurrence in Perl 6 rules. I think he might have posted directly to Google Groups, as I did not receive it in my email and nobody replied. The example Sudarshan gave matches the seconded occurrence. As I am not sure what Sudarshan meant by "2nd last", I will answer the question I do know. m:2nd/foo/ will match the second occurrence of foo. You can access the nth occurrence similarly.

Perl 6 Compilers

Once again it is a race to see whether someone posts a comment to, or Google picks up, Perl 6 Compilers first.

Parrot

Welcome to the week of the continuation. If continuations make your head hurt overly, I advise taking a few Advils before continuing. If continuations are the way you naturally think, I advise you to stop playing with LISP compilers.

String Pinning

Bernhard Schmalhofer wanted to pass the same C string into two different external functions so that the first could do things to it that the second required. However, the solution that he found kept eating his string after the first invocation. Dan and Leo concluded that using the b parameter for NCI is correct in this situation.

GC Invocation

Last week Leo posted a radical idea about GC invocation. I decided not to invoke Warnock's Dilemma too soon. This week, I invoke it.

perl vs. /usr/bin/perl

Gerd Pokorra requested that #! perl change to #! /usr/bin/perl. Although Warnock applied, I think this will likely not happen. Not all systems have Perl installed in /usr/bin/ and people might want to change which perl to invoke by changing their paths.

Command Line Support for Various Compilers

Will Coleda added a TODO item so that Parrot can take various compiler arguments on the command line. What are you waiting for?

Register Stomping

Matt Diephouse had troubles with his newly updated Forth compiler. The verdict seems to be that he has uncovered a bug. To quote Matt, "*sigh*."

Calling Continuations, Basic Blocks, and Register Allocation

Either Leo or Jeff Clites (I don't recall who) started what is probably the largest thread for this week, focusing on register allocation and continuation troubles. Apparently the presence of continuations can unduly extend the lifetime of registers (as can try-catch things, but they seem less important). The suggested solutions all have some shortcomings, although it appears that only using lexicals is winning.

Basic Compilation Example

Jeff Clites wondered how something like (a + b) would compile. The answer is to initialize an intermediate PMC to Undef, which will morph as appropriate.

Eval Changes

Leo updated the implementation of eval to make it even cooler, or possibly just clean it up and fix some bugs ... I forget.

Basic Operations for MMD

Dan asked for volunteers to spec out the basic vtable and MMD functions for CLS (common language specification-or-other). Sam Ruby said he would be willing to take a stab at it in a few weeks. One week has gone by and Sam grows ever closer to stabbing things.

No Limit on IMCC Identifier Length

Will Coleda noted that there was no limit on the length of IMCC identifiers. Here come the eval buffer overflow attacks....

Duplicate Local Labels Break PASM

Will noticed that duplicate local labels cause IMCC to output broken PASM. I could be mistaken, but I though that IMCC -> PASM has been broken for a while...

Parrot API

Leo wondered if the parrot_api.pl could be less strict about what it thinks is and is not public. Jarkko Hietaniemi responded that this was not his call to make, and silence reigned supreme.

Empty Sub Troubles

Will Colleda posted a bug for empty subs in IMCC. Stéphane Payrard provided a fix. Leo applied it.

Languages with Object Support?

Jeff Horwitz wondered if there were any languages with object support that he could bend to the evil ends of mod_parrot. While no one answered, I think Parakeet might be such a language....

WinXP Problems

Christian Lott has been having troubles with WinXP, in particular with ops2vim.pl. Read on for the blow-by-blow.

Strings + Working Correctly = Pain

In his noble efforts to make Parrot respect foreign languages and cultures, while not necessitating ICU, Dan has massively reworked strings. He reports ... progress. He also posed a question for all to ponder: "How global should the defaults be?"

runtime/parrot/library/config.pbc

Nicholas Clark wondered what would generate this file. Leo answered. Nicholas tracked his problem down to a quirk in FreeBSD make and fixed it.

#line Directives from PMC Files

Nicholas Clark announced his triumph over #line directives in pmc2c2.pl. People were impressed, but promptly asked how to turn it off. Kids these days don't appreciate what they have. ;-)

Tail Calls

Jeff Clites asked about tail calls in the presence of continuations. After much debate, the consensus seems to be that they don't affect each other much at all. In fact, continuations make tail calls easier. However, the topic was broached as to whether these were an optimization that IMCC should detect, or should compilers generate them explicitly. I think the final decision was compiler generated, although I doubt I would be alone in not minding if IMCC had a flag to allow it to add them as it saw fit....

Character Classes

Patrick R. Michaud wanted to know if Parrot had character class support (that is, checking membership). Leo responded that it would after Dan's aforementioned string stuff.

POSTCOMP vs. LOAD

Leo asked about the differences between POSTCOMP and LOAD. Dan provided the answer: POSTCOMP runs only when (and as soon as) the sub is compiled. LOAD runs when the sub is loaded. This need not be the same time if the source is used to generate bytecode that is later executed.

Register Allocation/Spilling/Volatility

Jeff Clites suggested using variable-size register frames with explicit, volatile and non-volatile sections. Leo responded that this would cause serious code bloat, and pointed him to his and Miroslav Silovic's "watermark" scheme. Others voiced support for this as well as removing the implicit operands. Dan said no.

AIX JIT Gnomes

Adam Thomason observed that the AIX JIT problems have gone away and thanked the gnomes responsible. Remember, the proper sacrifice for little people like gnomes are mini-Milanos or other reduced-size cookies (nothing from Keebler as the gnomes and elves have long-unresolved issues).

Register Allocation (Again)

Bill Coffman's register allocator (or should I say his implementation of the Matula/Chaitin/Briggs algorithm) grows ever closer to being committed (although my initial prediction of two weeks ago was clearly wrong). In fact, his algorithm is so good that it even turned up bugs from other pieces of the code. Dan started a test run on his evil code, so we should have truly stressed performance numbers within the next several fortnights.

IMCC Segfault on Bad if

Gopal V found a way to make IMCC segfault. I think it is still waiting for a fix....

No perldoc == BIG FAT WARNING

James deBoer submitted a patch to fail if the configuration step did not find perldoc last week. This week the request has gone out to modify it to warn in a big and preferably fat way. No answer yet.

RT Clean Up

Will Coleda, whose diligent minding of the RT queue is much appreciated, closed a few tickets that no longer apply.

BASIC Compiler

Will "Dr. Frankenstein" Coleda has attempted to resurrect the basic compiler with some major surgery. He is making progress. Joshua Gatcomb put out a request for a new maintainer.

Config Provides Slightly More Info

Luke Palmer provided a patch for making Configure.pl slightly more informative. Brent "Dax" Royal-Gordon applied it.

Find-Method Inheritance

Sam Ruby asked for comment before committing a patch to allow calling of attributes as methods and to allow inheritance of attributes. When no one commented, he committed.

Threads, Events, Win32

Gabe Schaffer pointed out some issues with threads and events from a Win32 perspective. Leo pointed out that this has not yet been fully implemented/abstracted/documented and observed that Gabe would be a welcome volunteer.

Core Dump on japh16

Bernhard Schmalhofer noticed that japh16 core dumps. He also submitted a simplistic fix.

The Usual Footer

If you find these summaries useful or enjoyable, please consider contributing to the Perl Foundation to help support the development of Perl. You might also like to

Cross-Language Remoting with mod_perlservice

Mod_perlservice? What is That?

Mod_perlservice is a cool, new way to do remoting -- sharing data between server and client processes -- with Perl and Apache. Let's start by breaking that crazy name apart: mod + perl + service.

Mod means that it's a module for the popular and ubiquitous Apache HTTP Server. Perl represents the popular and ubiquitous programming language. Service is the unique part. It's the new ingredient that unifies Apache, Perl, and XML into an easy-to-use web services system.

With mod_perlservice, you can write Perl subs and packages on your server and call them over the internet from client code. Clients can pass scalars, arrays, and hashes to the server-side subroutines and obtain the return value (scalar, array, or hash) back from the remote code. Some folks refer to this functionality as "remoting" or "RPC," so if you like you can say mod_perlservice is remoting with Perl and Apache. You can write client programs in a variety of languages; libraries for C, Perl, and Flash Action Script are all ready to go.

Now that you know what mod_perlservice is, let's look at why it is. I believe that mod_perlservice has a very clean, easy-to-use interface when compared with other RPC systems. Also, because it builds on the Apache platform it benefits from Apache's ubiquity, security, and status as a standard. Mod_perlservice sports an embedded Perl interpreter to offer high performance for demanding applications.

mod_perlservice

How Can I Use mod_perlservice?

Mod_perlservice helps create networked applications that require client-server communication, information processing, and sharing. Mod_perlservice is for applications, not for creating dynamic content for your HTML pages. However, you surely can use it for Flash remoting with Perl. Here are some usage examples:

  • A desktop application (written using your favorite C++ GUI library) that records the current local air temperature and sends it to an online database every 10 minutes. Any client can query the server to obtain the current and historical local air temperature of any other participating client.
  • A Flash-based stock portfolio management system. You can create model stock portfolios and retrieve real-time stock quote information and news.
  • A command-line utility in Perl that accepts English sentences on standard input and outputs the sentences in French. Translation occurs in server-side Perl code. If the sentence is idiomatic and the translation is incorrect, the user has the option of sending the server a correct translation to store in an online idiom database.

How Do I Start?

Let's move on to the fun stuff and set up a working installation. Before we begin, make sure you have everything you need! You need Apache HTTPD, Perl, Expat, mod_perlservice, and a mod_perlservice client library (Perl Client | C Client | Flash Client). You must download a client library separately, as the distribution does not include any clients! In your build directory:

myhost$ tar -xvzf mod_perlservice.tar.gz 
myhost$ cd mod_perlservice
myhost$ ./configure
myhost$ make
myhost$ make install

If everything goes to plan, you'll end up with a fresh mod_perlservice.so in your Apache modules directory, (usually /etc/apache/modules). Now it's time to configure Apache to use mod_perlservice. cd into your Apache configuration directory (usually /etc/apache/conf) Add the following lines to the file apache.conf (or httpd.conf, if you have only a single configuration file):

LoadModule perlservice_module modules/mod_perlservice.so
AddModule mod_perlservice.c

Add the following lines to commonapache.conf, if you have it and httpd.conf if you don't:

<IfModule mod_perlservice.c>  
<Location /perlservice>   SetHandler
mod_perlservice   
   Allow From All PerlApp
   myappname /my/app/dir
   #Examples
   PerlApp stockmarket /home/services/stockmarket   
   PerlApp temperature /home/services/temperature  
</Location>
</IfModule>

Pay close attention to the PerlApp directive. For every mod_perlservice application you want to run, you need a PerlApp directive. If I were creating a stock market application, I might create a directory: /home/services/stockmarket and add the following PerlApp directive:

PerlApp stockmarket /home/services/stockmarket

This tells mod_perlservice to host an application called stockmarket with the Perl code files located in the /home/services/stockmarket directory. You may run as many service applications as you wish and you may organize them however you wish.

With the configuration files updated, the next step is to restart Apache:

myhost$ /etc/init.d/apache restart
or
myhost$ apachectl restart

Now if everything went as planned, mod_perlservice should be installed. Congratulations!

An Example

Let's create that stock portfolio example mentioned earlier. It won't support real-time quotes, but will instead create a static database of common stock names and historical prices. The application will support stock information for General Electric (GE), Red Hat (RHAT), Coca-Cola (KO), and Caterpillar (CAT).

The application will be stockmarket and will keep all of the Perl files in the stock market application directory (/home/services/stockmarket). The first file will be quotes.pm, reading as follows:

our $lookups = {
    "General Electric" => "GE",
    "Red Hat"          => "RHAT",
    "Coca Cola"        => "KO",
    "Caterpillar Inc"  => "CAT"
};
our $stocksymbols = {
    "GE" => {
        "Price"            => 33.91,
        "EarningsPerShare" => 1.544
    },
    "RHAT" => {
		"Price" => 14.96,
		"EarningsPerShare" => 0.129
	},
    "KO"   => {
        "Price"            => 42.84,
        "EarningsPerShare" => 1.984
    },
    "CAT" => {
        "Price"            => 75.74,
        "EarningsPerShare" => 4.306
    }
};

package quotes;

sub lookupSymbol {
    my $companyname = shift;
    return $lookups->{$company_name};
}

sub getLookupTable {
    return $lookups;
}

sub getStockPrice {
    my $stocksymbol = shift;
    return $stocksymbols->{$stocksymbol}->{"Price"};
}
sub getAllStockInfo {
    my $stocksymbol = shift;
    return $stocksymbols{$stocksymbol};
}
1;

That's the example of the server-side program. Basically, two static "databases" ($lookups and $stocksymbols) provide information about a limited universe of stocks. The above methods query the static databases; the behavior should be fairly self-explanatory.

You may have as many .pm files in your application as you wish and you may also define as many packages within a .pm file as you wish. An extension to this application might be a file called news.pm that enables you to fetch current and historical news about your favorite stocks.

Now let's talk some security. As it stands, this code won't work; mod_perlservice will restrict access to any file and method you don't explicitly export for public use. Use the .serviceaccess file to export things. Create this file in each application directory you declare with mod_perlservice or you'll have no access. An example file might read:

<ServiceAccess>
  <AllowFile name="quotes.pm">
    Allow quotes::*
  </AllowFile>
</ServiceAccess>

In the stock market example, this file should be /home/services/stockmarket/.serviceaccess. Be sure that the apache user does not own this file; that could be bad for security. This file allows access to the file quotes.pm and allows public access to all (*) the methods in package quotes.

If I want to restrict access only to getStockPrice, I would have written Allow quotes::getStockPrice. After that, I could add access to lookupSymbol with Allow quotes::lookupSymbol. To make quotes.pm public carte blanche, use Allow *. You won't need to restart Apache when you make changes to this file as it reloads automatically.

Client Code

Well, so far I've only shown you half the story. It's time to create some client-side code. This client example uses the Flash "PerlService" library, just one of the client-side interfaces to mod_perlservice. The Flash client works well for browser interfaces while the Perl and C clients can create command-line or GUI (ie, GTK or Qt) applications. This article is on the web, so we'll give the Flash interface a spin and then go through an example in Perl.

The first code smidgen should go in the first root frame of your Flash application. It instantiates the global PerlService object and creates event handlers for when remote method calls return from the server. The event handlers output the requested stock information to the display box.

#include "PerlService-0.0.2.as"
// Create a global PerlService object
// Tell the PerlService object about the remote code we want to use:
// arg1) host: www.ivorycity.com
// arg2) application: stockmarket
// arg3) file: quotes.pm
// arg4) package: quotes
_global.ps = new PerlService("www.ivorycity.com","stockmarket","quotes.pm","quotes");
// First declare three callback functions to handle return values
function onStockPrice(val) { 
	output.text = "StockPrice: " + symbolInput.text + " " + val + "\n" + output.text;
}

function onAllStockInfo(val) { 
    output.text = "Stock Info: " + allInfoInput.text + "\n" + "\tPrice: "
	              + val.Price + "\n" + "\tEarnings Per Share: "
				  + val.EarningsPerShare + "\n" + output.text;
}

function onLookupSymbol(val) { 
	output.text = "Lookup Result: " + symbolInput.text + " " + val + "\n"
	              + output.text;
}

// Register callback handlers for managing return values from remote  methods
// ie, onStockPrice receives the return value from remote method getStockPrice

ps.registerReplyHandler( "getStockPrice", onStockPrice );
ps.registerReplyHandler( "getAllStockInfo", onAllStockInfo );
ps.registerReplyHandler( "lookupSymbol", onLookupSymbol );

Now for the code that makes things happen. The following code attaches to three separate buttons. When clicked, the buttons call the remote Perl methods using the global PerlService object. Flash Action Script is an event-driven system, so click event-handlers will call the remote code and return event-handlers will do something with those values.

buttons and code associations
Figure 1. Button and code associations.

When a user presses Button 1, call the remote method getStockPrice and pass the text in the first input box as an argument.

on (release) {
	ps.getStockPrice(box1.text);
}

When the user presses Button 2, call the remote method getAllStockInfo and pass the text in the second input box as an argument.

on (release) {
	ps.getAllStockInfo(box2.text);
}

When the user presses Button 3, call the remote method lookupSymbol and pass the text in the third input box as an argument.

on (release) {
	ps.lookupSymbol(box3.text);
}

That's the entire Flash example. Here is the finished product.

Perl Client

Not everyone uses Flash, especially in the Free Software community. The great thing about mod_perlservice is that everyone can join the party. Here's a Perl Client that uses the same server-side stock market API.


use PService;

my $hostname = "www.ivorycity.com";
my $appname  = "stockmarket";
my $filename = "quotes.pm";
my $package  = "quotes";              

#Create the client object with following arguments:      
#1) The host you want to use
#2) The application on the host
#3) The perl module file name
#4) The package you want to use

my $ps = PSClient->new( $hostname, $appname, $filename, $package );

# Just call those remote methods and get the return value
my $price  = $ps->getStockPrice("GE");
my $info   = $ps->getAllStockInfo("RHAT");
my $lookup = $ps->lookupSymbol("Coca Cola");                   

#Share your exciting new information with standard output
print "GE Price: " . $price . "\n";
print "Red Hat Price: " . $info->{Price} . "\n";
print "Red Hat EPS: " . $info->{EarningsPerShare} . "\n";
print "Coca-Cola's ticker symbol is " . $lookup . "\n";

Using the PSClient object to call remote methods might feel a little awkward if you expect to call them via quotes::getStockPrice(), but think of the $ps instance as a proxy class to your remote methods, if you like.

If things don't work, use print $ps->get_errmsg(); to print an error message. $ps->get_errmsg(); That's a local reserved function, so it doesn't call the server. It's one of a few reserved functions detailed in the Perl client reference.

As you can see, it requires much less work to create an example with the Perl client. You simply instantiate the PSClient object, call the remote methods, and do something with the return values. That's it. There is no protocol decoding, dealing with HTTP, CGI arguments, or any of the old annoyances. Your remote code may as well be local code.

Thanks for Taking the Tour

That's mod_perlservice. I'm sure many of you who are developing client-server applications can see the advantages of this system. Personally, I've always found the existing technologies to be inflexible and/or too cumbersome. The mod_perlservice system offers a clean, simple, and scalable interface that unites client-side and server-side code in the most sensible way yet.

What's next? mod_parrotservice!

TEST

This is a test

Again...

This Week in Perl 6, November 2 - 8 2004

Welcome to yet another summary, brought to you (once again) with the aid of the musical stylings of Dar Williams and Soul Coughing and a small, stuffed elephant name Aliya. And, without further ado, I give you Perl 6 Language (whose traffic has picked up a little).

Perl 6 Language

What Was that Anonymous Thing?

Juerd wondered what things could be named and what could be anonymous. Larry provided the answer: Subtypes, Enums, Lists (Lazy and Eager), Grammars, and Packages, but then threatened to attack Juerd with hot grits.

Updated Apocalypses and Synopses

Larry proved links to the current versions of the various Apocalypses and Synopses. Unfortunately he forgot to start a new thread with his message....

Suggested Warning for Overriding Operators

Aaron Sherman wanted to receive a warning for defining things like multi sub *infix:+(...) {...}. But Larry reasoned that anyone who'd use the * there does not care for warnings.

Perl 6 Compiler

Last week, I bemoaned my lack of Google (and thus links). But a couple kind souls pointed out that I could get links straight from the horses mouth at nntp.perl.org. All I can say is, ::shrug:: "Who knew that there was an Internet outside of Google?" Fortunately, I do not have to admit to its existence as there were no messages this week.

Parrot Internals

She-bangs for None!!

Last week James deBoer offered a patch to remove all of the shebang lines from config/*.pl (after his initial patch to add them to all of them was turned away). Warnock applies.

Solaris 9 Troubles

Christian Aperghis-Tramoni has some trouble while trying to install on Solaris 9. Warnock applies.

More Vtables

Leo wanted to add a finalize vtable entry (in addition to destroy, which is apparently used for free memory and not active resources). Jeff commented on the difficulties implicit in finalizing stuff, and the thread ran out of gas.

Build Dynclasses by Default

Last week, Leo was stalling to hear about success and failures before adding them to the default build. Brent "Dax" Royal-Gordon and Sam Ruby both chimed in with success. And so they did.

Register Frame Recycling

Leo added in some basic continuation recycling. Dan didn't like that it required the user to clone return continuations into full ones. Leo agreed, but felt that a returncc function would be needed first. I wondered why we needed this recycling and couldn't just let the DOD/GC do it for us. The answer: speed.

Upcase Binary Strings

Dan wondered what should happen if you tried to play with the case of a binary encoded string. The consensus seems to be either throw an exception or nothing, depending on settings.

Setref Poorly Named

Sam Ruby was initially confused by the strange behavior of setref. Leo told him that it was intended to set a reference inside a reference type and noted that classes needs a clean up. With the advent of dynclasses, this sounds like a job for some adventurous lurker...

Tracebacks PMC v. Ops

Leo wondered if we should have a PMC that could access the entire call chain and do whatever evil it wanted. Dan conjectured that this sort of thing was evil enough that ops might be well advised. Leo initially put some methods into the continuation to do this, but later thought about putting them into the interpreter instead. I like the interp idea.

Parrot -t Memory Leaks

Last week our fearless leader notice some significant memory leaks with parrot -t. This week our fearless pumpking fixed them.

Performance Graphs

Matt Diephouse (assisted by Joshua Gatcomb) provided a pointer to a page of periodic Parrot performances, provided as pretty pictures. Please provide possible improvement pointers.

Uniline yield() and return()

Stéphane Payrard (whose name Google objects to strenuously) re-sent his patch for uniline yield and return in PIR. Leo applied the patch.

Mod_parrot 0.1

Jeff Horwitz released mod_parrot 0.1. Pretty nifty.

download mod_parrot

IO Auto-Flush Troubles

Christian Aperghis-Tramoni wondered how to make stdout not buffer away his prompt. The answer (provided by Mary Pauley and Luke Palmer) requires using pioctl and strange magic numbers.

BigChanges--; release++?

Since the big indirect register change has gone through, some voices suggested putting out a 0.1.2 release. Leo felt that it was too soon, but seemed to be holding a minority opinion. I vote that we release after the new register allocator (and possibly Dan's string stuff depending on how long it takes).

Dan's String Stuff

Dan is about to make some "Irrevocable Changes" so he decided to do his work in a branch named pluggable_encodings, assuming he can use CVS (which is surprisingly hard at times). Because no one screamed foul, I think it is a safe assumption.

Clone mmd

Sam Ruby submitted a patch adding mmd_clone for dynclasses. Leo suggested an alternate approach, and Warnock['s Dilemma] took over.

True Coroutines

Klaas-Jan Stol wondered if Parrot had "true" coroutine because Roberto Ierusalimschy (of Lua fame) called it into question. Some confusion ensued as to what the platonic ideal of a coroutine actually is, but the answer seems to be: we should and if they aren't, patches welcome.

Trouble with Closures and Subs

Klaas-Jan Stol was having trouble with closures and subs. Many people provided some very useful pointers. If you are writing a compiler for Parrot you should read this thread carefully. If you are writing a summary for Parrot you should skim it and punt.

Shared Library and Dynclasses

Sam Ruby wondered if dynclasses should link to libparrot.so. Dan provided the answer: yes. Sam Ruby went on to suggest that if most PMCs would eventually become dynclasses, then (to reduce executable size, the Parrot executable should probably also link to libparrot.so rather than including all of it. No one commented on this observation.

Search Paths and Libraries

Dan wants to stop being annoyed by library path issues. He figures that the right way to do this is to allow dynamic library loading. Best of all he wants the code to do that be in the library. Thus he put out a request for the ability to embed bytecode into an executable. Brent "Dax" Royal-Gordon provided a slightly too-evil method of doing this. Other slightly less-evil (or even good) options would definitely be received thankfully.

Benchmarks? Tests? Or Both?

Joshua Gatcomb noticed that some benchmarks fail on occassion. Matt Diephouse suggested making them tests too. Leo felt that would be fine provided they did not take too long to run.

Fibonacci Still Slow

It is odd how the wheel comes full circle. Originally designed as an exercise in addition for students, the Fibonacci Sequence has some really interesting mathematical properties. While completely ignoring its unexpected niftiness (and explicit formula), this miraculous sequence is still used as busy work for poor creatures. This time the creature is Parrot, and despite Leo's best efforts, it is still slow. There was some talk of how to optimize this before Dan put the smack down all of it:

"No more performance changes. Period. We get Parrot fully functional first."

Parrot on Windows

Jerry Wiltz wondered how to get Parrot on his poor Win XP box. Fred and Georgy provided many pointers. Christian Lott also had trouble, but Nicholas Clark, Ron Blaschke, Peter Sinnott, and Steve Peters all provided pointers.

Parrot Build Issues

Jack J. Woehr was having trouble building parrot on his Sun. Leo provided the needed help. Then ICU became problematic. Many people provided help and he eventually got it running Forth, which promptly segfaulted. Jack sounded discouraged, but he really should stay with it now that he has overcome the hard part....

Detecting perldoc

James deBoer wondered if we should detect the presence of perldoc and fail to configure if it is missing. The consensus seems to be that we should detect the absence of perldoc, but if it is not there throw up a big fat warning and proceed.

Pow and Other Ops

There was/is a fairly long and ongoing discussion about ops on PMCs, which Sam Ruby unwittingly started when he asked for MMD pow. While there are people on all sides, it appears likely that the ops will stay.

C on Parrot?

Bloves wondered what the current state of C on Parrot was. But I think he posted it through Google Groups (which does not actually reach the list) as I did not find it in my email. The answer is it essentially non-existent, but if you are feeling brave/insane/suicidal patches welcome.

Streams and Filters

Ron Blaschke tracked does the problem with t/library/streams.t #14 on Win32. The guilty party is line endings. One workaround would be to set the file in question to binary in CVS (although that feels like cheating).

AIX PPC JIT

Adam Thomason provided a patch to fix some warnings. Leo applied it.

Register Allocation/Volatility

Despite Dan's line-in-the-sand strong statements about no more optimization, Jeff Clites and Leo tossed about some radical ideas for how to speed everything up.

GC Invocation

Leo suggested changing the GC from one that interrupts everybody and does its thing, to one that waits its turn and does its thing. I am concerned that this could lead to resource starvation in a multithreaded system, but nobody else posted anything. Of course, he only suggested this today.

Eval Questions

Leo wants some answers to some eval questions before he tries to clean up dynamic code compiling. Answers??

Include Perl 6 Files in Config

Gerd Pokorra wants to be able to run Perl 6 from anywhere. He even provided a means of achieving this end (although not in unified diff format as is the custom).

pioctl -> segfault

Stéphane Payrard received a segfault while playing with the previously mentioned pioctl magic numbers. Leo speculates that this might be due to the lack of an end opcode.

[TODO] Improve parrot-config.imc

Takers wanted.

Missing Arguments

Klaas-Jan Stol wondered how to detect missing arguments. Leo and Dan both pointed to the argument count registers.

Update to embed.pod

Stéphane Payrard provided an update to embed.pod. Leo applied it, but Brent "Dax" Royal-Gordon cried foul over the including of parrot.h. Dan agreed with Brent, and Stéphane provided a patch to remove the offending line.

Calling Conventions, Tracebacks, and Register Allocations

Leo suggested a new way to invoke functions that would clean up calling, tracebacks, and register allocation. While such a change would have great aesthetic value, Dan declared it premature as Parrot is not yet fully specified. Doubtless this will re-arise later, as aesthetics is a powerful motivator.

The Usual Footer

If you find these summaries useful or enjoyable, please consider contributing to the Perl Foundation to help support the development of Perl. You might also like to send feedback.

Implementing Flood Control

Accordingly to Merriam-Webster Online, "flood" means:

1: a rising and overflowing of a body of water especially onto normally dry land;

2: an overwhelming quantity or volume.

In computer software there are very similar situations when an unpredictable and irregular flow of events can reach higher levels. Such situations usually are not comfortable for users, either slowing down systems or having other undesired effects.

Floods can occur from accessing web pages, requesting information from various sources (ftp lists, irc services, etc.), receiving SMS notification messages, and email processing. It is obvious that it is not possible to list all flood cases.

"Flood control" is a method of controlling the processing-rate of a stream of events. It can reject or postpone events until there are available resources (CPU, time, space, etc.) for them. Essentially the flood control restricts the number of events processed in a specific period of time.

Closing the Gates

To maintain flood control, you must calculate the flood ratio, which is:

flood ratio equation
Figure 1. Flood ratio equation.

fr flood ratio
ec event count
tp time period for ec

To determine if a flood is occurring, compare the flood ratio to the fixed maximum (threshold) ratio. If the result is less than the threshold, there's no flood. Accept the event. If the result is higher, refuse or postpone the event.

comparing the ratios
Figure 2. Comparing the ratios.

ec event count
tp time period for ec
fc fixed event count (max)
fp fixed time period for fc

It is possible to keep an array of timestamps of all events. Upon receipt of a new event, calculate the time period since the oldest event to use as the current count/time ratio. This approach has two drawbacks. The first is that it uses more and more memory to hold all of the timestamps. Suppose that you want only two events to happen inside a one-minute period, giving two events per minute. Someone can trigger a single event, wait half an hour, and finally flood you with another 58 requests. At this point the ratio will be 1.9/min., well below the 2/min. limit. This is the second drawback.

A better approach is to keep a sliding window either of events (fc) or time period (fp).

This period window requires an array of the last events. This array size is unknown. (The specific time units are not important, but the following examples use minutes.)


               past                                   now
    Timeline:  1----2----3----4----5----6----7----8----9---> (min)
    Events:    e1      e2 e3         e4     e5 e6     e7

This timeline measures event timestamps. To calculate the flood ratio, you count events newer than the current time window of size fp. And check against a ratio of four events in three minutes:

Time now:      9
Time window:   from 9-3 = 6 to now(9), so window is 6-9
Oldest event:  e5 (not before 6)
Event count:   3 (in 6-9 period)
Flood ratio:   3/3

This ratio of 3/3 is below the flood threshold of 4/3, so at this moment there is no flood. Perform this check at the last event to check. In this example, this event is e7. After each check, you can safely remove all events older than the time window to reduce memory consumption.

The other solution requires a fixed array of events with size fc. With our 4ev/3min example, then:


               past                  now
    Timeline:  <--5----6----7----8----9---> (min)
    Events:      e4        e5 e6     e7

The event array (window) is size 4. To check for a flood at e7, we use this:

Window size "fc": 4
First event time: e4 -> 5
Last  event time: e7 -> 9
Time period "tp": 9-5 = 4
Flood ratio is:   4/4

The ratio of 4/4 is also below the threshold of 4/3, so it's OK to accept event e7. When you must check a new event, add it to the end of the event array (window) and remove the oldest one. If the new event would cause a flood, remember to reverse these operations.

If the flood check fails, you can find a point in the future when this check will be OK. This makes it possible to return some feedback information to the user indicating how much time to wait before the system will accept the next event:

time until next event equation
Figure 3. Time until next event equation.

ec  event count (requests received, here equal to fc)
fc  fixed event count (max)
fp  fixed time period for fc
now the future time point we search for
ot  oldest event time point in the array (event timestamp)

simplified time until next event equation
Figure 4. Simplified time until next event equation.

time to wait equation
Figure 5. The time-to-wait equation.

time the actual current time (time of the new event)
wait time period to wait before next allowed event

If wait is positive, then this event should be either rejected or postponed. When wait is 0 or negative, it's OK to process the event immediately.

The Code

In the following implementation I'll use a slightly modified version of the sliding window of events. To avoid removing the last event and eventually replacing it after a failed check, I decided to check the current flood ratio with the existing events array and with the time of the new one:


               past                  now
    Timeline:  <--5----6----7----8----9---> (min)
    Events:      e3   e4   e5 e6    (e7)

    Window size fc: 4 (without e7)
    First event time: e4 -> 5
    Last event time: e7 -> 9
    Time period tp: 9-5 = 4
    Flood ratio is:   4/4

This seems a bit strange at first, but it works exactly as needed. The check is performed as if e6 is timed as e7, which is the worst case (the biggest time period for the fixed event window size). If the check passes, than after removing e3, the flood ratio will be always below the threshold!

Following this description I wrote a function to call for each request or event that needs flood control. It receives a fixed, maximum count of requests (the events window size) and a fixed time period. It returns how much time must elapse until the next allowed event, or 0 if it's OK to process the event immediately.

This function should be generic, so it needs some kind of event names. To achieve this there is a third argument -- the specific event name for each flood check.

Here is the actual code:

# this package hash holds flood arrays for each event name
# hash with flood keys, this is the internal flood check data storage
our %FLOOD;

sub flood_check
{
  my $fc = shift; # max flood events count
  my $fp = shift; # max flood time period for $fc events
  my $en = shift; # event name (key) which identifies flood check data

  $FLOOD{ $en } ||= [];   # make empty flood array for this event name
  my $ar = $FLOOD{ $en }; # get array ref for event's flood array
  my $ec = @$ar;          # events count in the flood array
  
  if( $ec >= $fc ) 
    {
    # flood array has enough events to do real flood check
    my $ot = $$ar[0];      # oldest event timestamp in the flood array
    my $tp = time() - $ot; # time period between current and oldest event
    
    # now calculate time in seconds until next allowed event
    my $wait = int( ( $ot + ( $ec * $fp / $fc ) ) - time() );
    if( $wait > 0 )
      {
      # positive number of seconds means flood in progress
      # event should be rejected or postponed
      return $wait;
      }
    # negative or 0 seconds means that event should be accepted
    # oldest event is removed from the flood array
    shift @$ar;
    }
  # flood array is not full or oldest event is already removed
  # so current event has to be added
  push  @$ar, time();
  # event is ok
  return 0;
}

I've put this on the CPAN as Algorithm::FloodControl.

To test it, I wrote a simple program that accepts text, line by line, from standard input and prints each accepted line or the amount of time before the program will accept the next line.

#!/usr/bin/perl
use strict;
use Algorithm::FloodControl;

while(<>)
  {
  # time is used to illustrate the results
  my $tm = scalar localtime;
  
  # exit on `quit' or `exit' strings
  exit if /exit|quit/i;
  
  # FLOOD CHECK: allow no more than 2 same lines in 10 seconds
  # here I use the actual data for flood event name!
  my $lw = flood_check( 2, 10, $_ );
  
  if( $lw ) # local wait time
    {
    chomp;
    print "WARNING: next event allowed in $lw seconds (LOCAL CHECK for '$_')\n";
    next;
    }
  print "$tm: LOCAL  OK: $_";
  
  # FLOOD CHECK: allow no more than 5 lines in 60 seconds
  my $gw = flood_check( 5, 60, 'GLOBAL' );
  
  if( $gw ) # global wait time
    {
    print "WARNING: next event allowed in $gw seconds (GLOBAL CHECK)\n";
    next;
    }
  print "$tm: GLOBAL OK: $_";
  }

I named this floodtest.pl. The of the test were: (">" marks my input lines)

cade@aenea:~$ ./floodtest.pl 
> hello
Wed Feb 17 08:25:35 2004: LOCAL  OK: hello
Wed Feb 17 08:25:35 2004: GLOBAL OK: hello
> hello
Wed Feb 17 08:25:38 2004: LOCAL  OK: hello
Wed Feb 17 08:25:38 2004: GLOBAL OK: hello
> hello
WARNING: next event allowed in 5 seconds (LOCAL CHECK for 'hello')
> bye
Wed Feb 17 08:25:43 2004: LOCAL  OK: bye
Wed Feb 17 08:25:43 2004: GLOBAL OK: bye
> hello
Wed Feb 17 08:25:45 2004: LOCAL  OK: hello
Wed Feb 17 08:25:45 2004: GLOBAL OK: hello
> see you
Wed Feb 17 08:25:48 2004: LOCAL  OK: see you
Wed Feb 17 08:25:48 2004: GLOBAL OK: see you
> next time
Wed Feb 17 08:25:52 2004: LOCAL  OK: next time
WARNING: next event allowed in 43 seconds (GLOBAL CHECK)
> one more try?
Wed Feb 17 08:26:09 2004: LOCAL  OK: one more try?
WARNING: next event allowed in 26 seconds (GLOBAL CHECK)
> free again
Wed Feb 17 08:26:31 2004: LOCAL  OK: free again
WARNING: next event allowed in 4 seconds (GLOBAL CHECK)
> free again
Wed Feb 17 08:26:42 2004: LOCAL  OK: free again
Wed Feb 17 08:26:42 2004: GLOBAL OK: free again

You can see that I could not enter "hello" 3 times during the first 10 seconds but still I managed to enter one more "hello" a bit later (the 10-second flood had ended for the "hello" line) and 2 other lines before the global flood check triggered (5 lines for 1 minute). After 60 seconds, floodtest.pl finally accepted my sixth line, "free again."

The next sections show how to use flood control in several applications. These examples are not exhaustive but are very common, so they will work as templates for other cases.

My Scores Please?

Imagine an IRC bot (robot) which can report scores from the local game servers. Generally this bot receives requests from someone inside IRC channel (a chat room, for those of you who haven�t used IRC) and reports current scores back to the channel. If this eventually becomes very popular, people will start requesting scores more frequently than it is useful just for fun, so there's a clear need for flood control.

I'd prefer to allow any user to request scores no more than twice per minute, but at the same time I want to allow 10 requests total every two minutes:

sub scores_request
{
    my $irc     = shift; # the IRC connection which I communicate over
                         # this is a Net::IRC::Connection object
    my $channel = shift; # the channel where "scores" are requested
    my $user    = shift; # the user who requested scores

    # next line means: do flood check for $user and if it is ok, then
    #                  check for global flood. this is usual Perl idiom.
    my $wait = flood_check( 2, 60, $user ) || flood_check( 10, 120, '*' );
    if( $wait ) # can be 0 or positive number so this check is simple
      {
      # oops flood detected, report this personally to the user
      $irc->notice( $user, "please wait $wait seconds" );
      }
    else
      {
      # it is ok, there is no flood, print scores back to the channel
      $irc->privmsg( $channel, get_scores() );
      }
}

This code uses the Net::IRC module, so if you want to know the details of the notice() and privmsg() functions, check the module documentation.

This is good example of combining events, but it works correctly only if the second flood ratio (in this case 10/120) is greater than first one (2/60). Otherwise you should extend the flood_check() function with an array of events to check in one loop, so if any of them fails the internal storage will update. Perhaps Algorithm::FloodControl will have such a feature in the future.

Another common case is to limit the execution of resource-consuming web scripts (CGI).

(Don't) Flood the Page!

If you want to limit CGI-script execution you will hit a problem: you must save and restore the flood-control internal data between script invocations. For this reason the Algorithm::FloodControl module exports another function called flood_storage, which can get or set the internal data.

In this example I'll use two other modules, Storable and LockFile::Simple. I use the first to save and restore the flood-control data to and from disk files and the second to lock this file to avoid corruptions if two or more instances of the script run at the same time:

#!/usr/bin/perl
use strict;
use Storable qw( store retrieve );
use LockFile::Simple qw( lock unlock );
use Algorithm::FloodControl;

# this is the file that should keep the flood data though /tmp is not
# the perfect place for it
my $flood_file = "/tmp/flood-cgi.dat";

# this is required so the web browser will know what is expected
print "Content-type: text/plain\n\n";

# I wanted to limit the script executions per remote IP so I have to
# read it from the web server environment
my $remote_ip = $ENV{'REMOTE_ADDR'};

# first of all--lock the flood file
lock( $flood_file );

# now read the flood data if flood file exists
my $FLOOD = retrieve( $flood_file ) if -r $flood_file;

# load flood data into the internal storage
flood_storage( $FLOOD ) if $FLOOD;

# do the actual flood check: max 5 times per minute for each IP
# this is the place where more checks can be done
my $wait = flood_check( 5, 60, "TEST_CGI:$remote_ip" );

# save hte internal data back to the disk
store( flood_storage(), $flood_file );

# and finally unlock the file
unlock( $flood_file );

if( $wait )
  {
  # report flood situation
  print "You have to wait $wait seconds before requesting this page again.\n";
  exit;
  }

# there is no flood, continue with the real work here
print "Hello, this is main script here, local time is:\n";
print scalar localtime;
print "\n...\n";

There are various issues to consider, such as the save/restore method, time required, and locking, but in any case the scheme will be similar.

Beep, Beep, Beep ...

In this last example I'll describe a small program, a variation of which I use for (email) SMS notifications. I wanted to avoid scanning large mail directories so I made my email filter copy incoming messages into a separate folder. The program scans this copy folder every 10 minutes for new messages. If there are any, it sends a notification for each one to my mobile phone and removes the copy of the message.

#!/usr/bin/perl
use strict;
use Algorithm::FloodControl;

our $MAIL_ROOT = '/home/cade/mail';
our @SCAN = ( 
              { # this is my personal mail, I'd like to be notified often
                FOLDER  => 'Personal2', # directory (mail folder) to scan
                FC      => 20,          # fixed event count
                FP      => 60*60,       # fixed time period, 1 hour
              }, 
              { # this is a mailing list, I don't need frequent notifications
                FOLDER  => 'AList2',    # directory (mail folder) to scan
                FC      => 3,           # fixed event count
                FP      => 20*60,       # fixed time period, 20 minutes
              }
            );
while(4)
  {
  process_folder( $_ ) for @SCAN;
  sleep(10*60); # sleep 10 minutes
  }

sub process_folder
{
  my $hr     = shift; # this is hash reference
  my $fc     = $hr->{ 'FC' };
  my $fp     = $hr->{ 'FP' };
  my $folder = $hr->{ 'FOLDER' };
  
  my @msg = glob "$MAIL_ROOT/$folder/*";
  return unless @msg; # no messages found
  for( @msg )
    {
    # there are new messages, so flood check is required
    my  $wait = flood_check( $fc, $fp, $folder );
    if( $wait )
      {
      # skip this pass if non-zero wait time is received for this folder
      print "FLOOD! $wait seconds required.\n";
      return;
      }
    send_sms( $folder, $_ );
    }
}

sub send_sms
{
  my $folder = shift;
  my $file   = shift;
  # implementation of this function is beyond the scope of this example
  # so I'll point just that it extracts subject line from the message file
  # and sends (over local sms gateway) text including folder name, time 
  # and subject
  unlink( $file );
  print "SMS: $folder, $file\n";
}

As you can see, this code -- while implementing a totally different task -- has exactly the same flood check as in the previous two examples.

Conclusion

I said in the beginning that flood control has a vast field of applications. There are many cases where it is appropriate or even necessary. There is no excuse to avoid such checks; implementing it is not hard at all.

This Fortnight on Perl 6, October 2004 Part Two

Perl 6 Summary for 2004-10-18 through 2004-10-23

Last week, I received a request to summarize perl6-language before internals. Frankly, it seems like a reasonable idea. Perl6-internals has always been first as long as I can remember. So perhaps, it is time to switch it up. Before I start off with perl6-language, let me remind all American readers to vote. Non-American readers should also vote, but it seems unlikely that your day of voting is tomorrow (as I write this). So without further ado...

Perl6 Language

Packrat Parsing

Thomas Sandlass posted a link to an article on Packrat Parsing. It looks like a promising techinque for small programs, although its memory requirements may be a bit heavy for large programs. Mr. Sandlass appears to have been Warnocked, but there are several explanations. First, he posted from Google; there has been trouble in the past about such things getting through. Second, he posted to Perl6 Language instead of Perl6 Compiler.

Assorted Questions

Rich Morin wanted to know two things. One, can he hook into function calls and just about everything else for his evil purposes? Two, can he get floating point values which automagically track their rounding errors? Austin Hastings pointed out that two would probably best be implemented as an opaque object whose value is a junction. Luke Palmer demonstrated an evil way to embed whatever hooks he wanted by adding an extra layer to the grammar. Somehow, I worry that shooting myself in the foot with this would take most of my body off...

Perl 6 Backwards Compatibility

 U. Ruirarchzatrea expressed a desire for perfect backwards
compatibility to Perl 5 and a Ponie too.  Matthew Walton, Brent "Dax"
Royal-Gordon, and Luke Palmer all told him that he could have a Ponie, but it
would not be perfect.

Perl6 Compiler

Perl6 Compiler List Troubles

In the last summary, I mentioned that Perl6 Compiler has not yet found its way to Google groups. Leo seems to have had trouble subscribing to it also. Perhaps this bears investigation? Does anyone know someone who can be poked/prodded/pleaded with about this? End result: no links for you!

Status Check?

Ovid popped in to see if he could find more detailed status info about the progress of P6C. Patrick Michaud provided a slightly unsatisfying answer of "not much longer," and Luke Palmer provided a slightly more detailed answer, which can be summarized as "not much longer."

Parrot

Unununium

Dan and Steve Fink told Jacques Mony that a port to Unununium would probably require substantial changes to Configure.

grsecurity Interfering with the Parrot JIT/build

Apparently, our current scheme of mem_alloc_executable and mem_free_executable is not quite enough to make grsecurity happy. Work is ongoing...

signal.t Failure

Will Coleda had problems with signal.t failing while his machine was under load. Leo pointed out that this is known behavior. Jeff Clites was a little confused and disappointed that his earlier patches to help solve this problem had not sufficed.

New Naming for parrot_interp

Brent "Dax" Royal-Gordon warned everyone that he was committing a gianormous change that would affect any pending patches. From the lack of chaos on the list, I would say that his warning worked.

Parrot Forth

Michel Pelletier and Matt Diephouse discussed some of the finer points of Forth implementations and optimizations.

Win32 Update

Ron Blaschke gave a quick update on his progress with VC7.1 on Win32. Looks good. Keep on chugging, Ron.

Downloading Parrot

Jerry Wiltz asked if help in downloading Parrot [he was starting from scratch (no Perl or C compiler)] on a WinXP box. Fred the LastNamelessOne provided a plethora of useful links.

The Return of Register Stacks

Miroslav Silovic posted a summary of a design change that he and Leo were considering. I asked for more details as to how it was different then something we had moved away from much earlier. Leo provided quite a nice and thorough explanation. Thanks. Dan observed that this recurring thread happened nearly yearly, but that the cycle was not quite 12 months. Dan also observed that the timing coinciding with his being sick was fortuitous. Leo and Dan went back and forth for a while discussing the implementation and implications of it. I believe that the end result is that we will have indirect access to registers; we will not need to have saveall/restoreall pairs around function calls; and Dan should eventually get better, but not until this is fully thrashed out.

JIT Rework

As a side effect of indirect access to registers, the JIT needs to be rejiggered to account for this. Leo and Jeff Clites went back and forth working on this with what sounds like good progress being made.

Win32 perlnum Test Fix

Ron Blaschke fixed a test failing because of -0.0. Leo applied the patch.

x86-64 Problems Fix

Brian Wheeler submitted a patch to fix x86-64. Leo couldn't apply it. Brian resubmitted. It got mangled on the way. Brian reresubmitted. Leo applied it. Thanks for perservering, Brian!

External Dependencies

Robert Spier suggested several options on how to simplify the problem of Parrot and external dependencies. Sadly, Warnock applies.

mod_parrot

Jeff Horwitz is making amazing progress with his cybernetically enhanced parrot. Oh wait, I mean embedding Parrot into Apache. It all looks really cool, and he has been able to use his experience to provide valuable critiques to us all.

Python, Parrot, and Scoping

Sam Ruby has "been trying to make sense of Python's scoping in the context of Parrot." Leo, Dan, and Allen Short all rushed to his aid.

Resizable*Array Implementation

Bernhard Schmalhofer wondered if the Resizable*Arrays should use chunked allocation. Dan told him, "No, the motivation behind them is simple and fast, thus the chunking would be unwanted overhead." He did mention that they could do something clever (like chunking) if they got big enough.

Register Spilling

In a flash of good news for Dan, Bill Coffman has started to put his mighty brain to work on the register spilling code. He, Leo, and Jeff discussed various aspects of it. I eagerly look forward to the results of getting Parrot a really cool spilling algorithm (although probably not a eagerly as Dan and his nasty pathological code ;-)

C89 Spec

Bill Coffman wondered if he could find the C89 spec so that he could ensure his changes were compliant. Dan suggested using K&R second edition. Jeff Clites pointed out the -std=c89 flag for gcc. Michael G Schwern suggested using C99 spec (which should also contain the C89 spec) as a "frightening accessory this Halloween." Michael frightens me.

method_util.c

Leo asked what the state of the functions in src/method_util.c is. He got Warnocked. I would say that means they should get pitched and see what happens.

.include 'absolute path'

Matthias Huerlemann created a patch allowing absolute paths in .include statements. Leo responed that the implementation was not quite right, but the format of the patch was good.

ICU + Red Hat == Pain

Joshua Gatcomb had a fairly long conversation with himself fleshing out a problem with ICU on Red Hat. Leo broke into his monologue to say that it seemed Red Hat specific.

Strange Segfault Involving String Concat

Matthias Hoelzl has a fairly hard-to-pin-down segfault. Leo suggested trying to use parrot -G, --gc-debug, and -t to track it down. No response yet.

pmccache

Leo suggested a small Perl task of making something akin to ccache for PMCs. Jeff Clites jumped on it.

Befunge Shall Live Again!!

Stepan Roh fixed somethings with the Befunge interpreter while simultaneously accusing befunge of being unimportant. Jerome applied some of those fixes and admonished Stepan for saying such things. I remember a time when Befunge fleshed out many troublesome bugs in Parrot, and I forsee a future when Befunge will be included in webservers everywhere through the magic of mod_parrot!

Siva, the multifaceted PMC

Stéphane Payrard asked about some problems he had encountered creating the magical all-in-one Siva PMC. Leo provided a helpful prod towards LVALUE_CAST. The other problems may or may not related to the Mandrake Cooker. Hopefully, someone will tell us.

JITted Functions Calling External Functions

Leo reported that, thanks to Jeff, one can now call into Parrot functions from JIT. Yay!

make testexec

Currently, make testexec will create and run a trivial "hello world" program for the exec core. Leo would like it to run the full test suite instead. This is a big thing but would probably win you many friends and even influence people.

Calling Conventions and Indirect Register Access

Leo provided several warnings that he was going to make some big changes that would break any code that did not follow the calling conventions. He went on to implement and commit a whole lot of stuff. I was impressed, and Dan later wondered if it was all done or not (as he wants to make some big changes soon). Apparently, it is for the most part.

Stalk Walking Bug

Nicholas Clark, with Leo's help, found a bug in the stalk walking code. Apparrently, our mask for valid pointers was a little overzealous. Nicholas checked in a quick fix. Steve Fink has a few suggestions.

Register Allocation

Bill Coffman is continuing his amazing work with register allocation. He has been making strong progress, and his new code is fast approaching production-ready. Unfortunately for Dan, it still has a few rough points and a small memory leak preventing it from being used on Dan's evil code. Leo has been helping him along the entire time, and I predict that all of Bill's work will find its way into Parrot before the next summary.

libjit

Robert Spier pointed out the fairly nifty libjit. Unfortunately, according to Leo, the current jit is far enough along that it would be too much work to switch. I tend to believe him.

Looking for a Hardware Donation

H. Merijn Brand reminded Leo that he hadn't been contacted yet about some "Diana" menthol alcohol. I was confused, but apparently Leo wasn't...

PMC/inode analogy

Stéphane Payrard wanted to know if his analogy between pmcs and filesystems was correct. Leo felt that it was mostly right and attempted to explain the value/variable split. Dan corrected Leo and went a little further. What I have taken away from this is: "do not think of it in terms of high level language operators, instead think of it in terms of C. You get pointers and values (and you can point to a value). Some things do shallow copies (i.e. pointer copies) others do deep."

pmc_type

Nicholas Clark wanted to know what a pcm_type of 0 means. The answer: no such type. This led to Stéphane Payrard asking about pmc_type values for abstract types. They don't get types because they can't be directly instantiated. This led Paolo Molaro to wonder about the virtue of having a system closer to C++ with array lookup for virtual functions. Nicholas, Leo, and Paolo banged on this for awhile and decided that there was some room for optimization. Leo scares me when he optimizes...

extend.c: Parrot_call

Leo was bothered by the fact that Parrot_call only supported void return values. Much to Jeff Horwitz's joy, he fixed it.

Computed Goto Destroyer of Weak Machines!!!

Vijay D. though that parrot was failing to install when it "hung" while compiling the computed goto cores. Nick Glencross, Leo, Dan, Peter Sinnott, and Joshua Gatcomb all helped out, explaining that cg is really hard on one's system and can be disabled with the --cgoto=0 flag to Configure.pl.

Byte Code

Peter Hickman wanted to know if Perl 6 would be able to be distributed in bytecode. The answer is yes, but this won't help you hide your source, as that will frequently be necessary meta-data. It will, however, speed up your compilation and even allow mmap of the bytecode to share between threads.

Pirate/Python

Sam Ruby has continued his work on Python and is making good progress. He was a little concerned about the speed issues of converting everything to dynclasses. He and Leo talked out some alternatives. Things are looking promising.

Threading Issues

Sam Ruby wanted to know about possible threading issues. Leo requested that they let the sleeping giant lie for a while longer.

.include "../relative path"

Matt Diephouse added a todo item for .includeing with relative paths. Takers welcome.

Array Fun

Bernhard Schmalhofer added a few benchmarks for the various array PMCs. Leo applied the patch.

Prederefed Run Cores

Leo had some concerns about the prederef run cores. Dan suggested pitching them, but Leo opted to fix them instead. Some discussion of their relative values ensued, and then someone started using assembly to support a point and my head exploded.

Parakeet in CVS

Michel Pelletier has finally gotten himself a perl.org account. The idea is that this would allow him to maintain Parakeet in CVS. Unfortunately, Warnock applies.

GC Crash

Matt Diephouse found a way to make Parrot's GC crash. Leo fixed it with alacrity.

PPC JIT failure

Jeff Clites found and fixed a bug in the grow-the-arena threshold for JIT. Leo seemed to feel that Jeff's solution was only a temporary one.

AIX PPC JIT warning

Jeff Clites voiced some concern over the state of JIT on AIX. Adam Thomason told him, "Worry not, it's already broken."

Call Chain Access

Appropriately, Leo found himself playing with chains around Halloween. He wondered if the call chain should have a PMC wrapper. Dan figured not, as ops which do evil things would be unlikely to be invoked accidentally, whereas a PMC might...

newsub opcodes

Leo wanted to add some newsub opcodes to save allocations. Dan countered that it would be a better thing to make constant PMCs work properly. Leo did.

pmc2c2.pm features

Leo suggested a couple of extra features that he wanted in pmc2c2.pm. Sam Ruby supplied them.

Build Dynclasses by Default

Will Coleda provided a patch to build dynclasses by default. Leo is stalling before applying it, hoping to hear from other platforms. Now would be a great time to chime in... ::cricket noises:::

Warnings for Duplicate Vars

IMCC currently doesn't produce them. Will Coleda supplied a naive attempt at this, but Leo decided it would be better to do it the elegant way and split the PASM and PIR lexers/parsers apart.

Parrot on AMD64

Brent "Dax" Royal-Gordon managed to find an AMD64 for himself. Not only am I jealous, but Parrot passes all expected tests on it when built with --ccflags=':add{ -fPIC }.

dynopslib Troubles

Klaas-Jan discovered that dynopslib was failing. Leo concurred and asked for patches.

makefile bad $(TOUCH)

Jeff Clites noticed that Makefile's $(TOUCH) does not actually work on many platforms and suggested a solution. Michael G. Schwern suggested using ExtUtils::Command instead. Who knows what will finally happen?

Debug ops

Dan wondered about adding some ops for debugging parrot code. The response was favorable and enthusiastic.

parrot -t Massive Leaks

Dan noticed that parrot -t leaks more memory then most cheap computers have.

hash vtable

Sam Ruby wondered about adding a hash vtable method for Python's benefit. Dan seemed to think it was a good idea.

disassembler broken

Matt Diephouse noticed that the dissambler was broken. Dan pointed out that he was testing it wrong, but Matt soon discovered that it didn't work even if he tested it right.

Shebangs for Everybody

James deBoer provided a patch adding shebang lines to config/*.pl. Brent "Dax" Royal-Gordon reasoned that these were unnessecary as these scripts should only be run by Configure.pl. James countered by offering a patch removing all of the shebang lines from the other ones. I like his style.

The Usual Footer

If you find these summaries useful or enjoyable, please consider contributing to the Perl Foundation to help support the development of Perl. You might also like to send feedback to ubermatt@gmail.com.

Komodo 3.0 Review

There are several different types of programmers, but I would wager that the vast majority are just like me: their mouse is a crucial input extension, and having an environment that supports the task at hand makes coding easier and, dare I say, more fun. This is where a GUI IDE (Integrated Development Environment) can help, with syntax highlighting, project management, code completion, reference documentation, support tools, and being able to use your mouse to place your cursor or to select a particular segment of code.

I'm a GUI IDE kind of guy, and I've been through quite a few of them to find the one that best suits me as a Perl Web Developer. ActiveState's Komodo (version 3.0.1) fits the bill, though there remains room for improvement. Let's start with the positive, and then I'll address where Komodo can improve.

Pluses

The last time that Perl.com reviewed Komodo was October 2002. At that time, Komodo was at version 2.0, which was slow and whose functionality seemed only halfway implemented. It showed promise then and has made huge leaps in subsequent versions to overcome the major failings, with speed increases, functionality improvements, and adding even more features.

My first criteria for any editor is (accurate) syntax highlighting. All editors have this to some extent, but when you throw some wacky Perl stuff their way (heredocs, for example), they start to choke and mislead you. Komodo has handled everything I've thrown at it flawlessly and has a neat preferences panel where you can customize the color scheme with your own flair (or some might say lack thereof when they see my screen).

Another important feature for an IDE is project management, a way to group a list of files relevant to the project you're working on in an easy-to-access fashion. Komodo has a Projects tab, through which you can do pretty much anything in that regard; it even includes a powerful Import from File System that'll help you set up a project in no time. Also, within that same tab is a Code Explorer, which will parse and index the open file for subroutines

Simon referred to Komodo's Toolbox in the previous review, and that's certainly been a great feature for me. I've setup some tools, such as PerlTidy, to run on an open file. Then, when I open a messy file, I can just double-click on that tool, and it'll clean up the file. I also have other tools to insert standard snippets of HTML and HTML::Template code. Snippets are really cool because you can set the cursor placement after you've inserted the snippet, as well as maintain the indentation styles. You can also assign Key Bindings to pretty much anything, including the tools. When I hit Ctrl-1, the editor will insert this:

<!-- TMPL_VAR NAME="" -->

into my code and put the cursor right there in the NAME parameter, so I can keep typing. I've included some instructions after the review below to show you how to set these up.

Simon also referred to the Rx Toolkit, which is still available and great to use to play around with regular expressions and figure out how they will work. Version 3.0 brings vast improvements to the Rx Toolkit, with different modes for Match, Match All (for multiline text to match against), Split, and Replace/Replace All. This tool is indispensable for those who are tinkering with a regular expression and trying to figure things out. ActiveState went a step further and added a Perl Interactive Shell that allows you to type or paste in lines of code and play around. This has been useful for me when figuring out data structures and how to access deeply-nested data.

Komodo also supports CVS [and Perforce (version control)], such that within the IDE, you can perform versioning functionality (diff, commit, checkout, etc). It will also monitor the files and repository so that if there are any changes, the files' indicator will change accordingly, keeping you aware of what's going on. You can also setup the CVS remotely over SSH to interact with a remote repository securely.

I feel like I've barely scratched the surface with what Komodo can do, both in this review as well as my personal experience with the IDE. Komodo has a powerful debugger, which in an IDE setting is a great way to visualize your code executing, with the line of code in one window and a tab with call stacks, output, watched variables (that you can change midstream if you like), breakpoints, and more. Komodo presents complex watched variables in a nested tree format, allowing you to expand elements of hashes and arrays to go through the data. Komodo also supports remote debugging, allowing you to debug a remote console script or web application right inside Komodo, stepping in and out of code and watching variables right off the remote server. Personally, I could never make this work (despite ActiveState Technical Support's best efforts), but I haven't had the need for it in the first place.

Komodo also supports different languages besides Perl, such as HTML, PHP, Python, and XML (including syntax highlighting support for 30 other languages). I do dabble in PHP from time to time, so Komodo gives me a one-stop IDE to handle that, too. I can also incorporate all of these file types into one project for easy access.

Just as the power of Perl comes from its community, ActiveState also extends a sense of community with Komodo, with ASPN, a portal of sorts that encompasses mailing lists, documentation, cookbooks (code recipes), and a bugzilla. With an ActivePerl Pro Studio subscription, ASPN also integrates with Safari, such that you can search and read over 1,000 technical books from O'Reilly, Addison Wesley, and other publishers.

Komodo installs a ton of helpful documentation and tutorials locally. You can easily pull up the documentation for a specific function/module by highlighting it and hitting Shift-F1 (which does a Google search on perldoc.com for the highlighted string) or Ctrl-F1 (which does a Google search on any site for the highlighted string). Komodo also comes with some sample tools in the Toolbox (that you could bind to another key combination) to run perldoc locally on the highlighted function or module.

ActiveState is also working on a new point release that will bring AutoComplete and CallTips to the Perl language, which will provide even more efficiency to coding and reference documentation. This next release will also address a few bugs (some of which I address below) and reinforces their dedication to responding to our feedback and pushing for even more functionality.

Minuses

Komodo 3.0 has a few hiccups and flaws, though. It seems the winds of version control have shifted to something called Subversion. The IDE doesn't directly support it, though you could certainly create supporting tools in the Toolbox. As I said earlier, I also had problems with Remote Debugging, though it was never a high-enough priority for me to figure out what was going on. Komodo also allows you to access files through FTP, but not SCP/SSH, which is more secure and becoming the standard for accessing remote files.

Also, when Komodo went from 2.5.2 to 3.0, they moved the Toolbox from a tab in the Projects Tab to a tab of its own, which kind of annoys me because it eats into my precious screen real estate. It would be nice to be able to customize the placements of the various tabs and what they contain. Komodo also doesn't support code descriptions in the Code Browser for Perl (they do for PHP and Python), though that may change, hopefully with the next release.

Lastly, there seem to be quite a few minor GUI flaws. There's nothing major to prevent you from doing your work; the flaws are things like double-clicking on a subroutine name in the Code Browser goes to the line, but in a huge Perl file, it doesn't center on the screen (you may have to scroll a bit) and more seldom, doesn't go to the right line (the start of the subroutine). I also had a problem with copy and pasting stuff in a line-wrapped code that would then appear as if I have lost code. Reflowing the paragraphs sometimes doesn't work (it would reflow more narrowly than it should have). When I ran the PerlTidy tool on my open file, I wouldn't see the "File Changed on Disk" alert until I Alt-Tabbed to another application and then Alt-Tabbed back.

Conclusion

The great thing about ActiveState is that their developers and technical support use their mailing lists and a Bugzilla to interact with the feedback and to track down these types of flaws and address them.

At the risk of sounding like a fanboy, ActiveState's Komodo is an amazing platform, surrounding you with the power to accomplish any Perl task at hand. Despite its negative points, they aren't enough to overcome the benefits it brings to the table, and I highly recommend it for Perl programmers of any variety. Komodo is available for Windows, Linux, and Solaris. Prices start at $29.95 for the personal edition (non-commercial/educational use) and $295 for the professional edition. Subscriptions, upgrade pricing, and a free 21-day fully-functional edition are also available off of their web site.

Toolbox Setup Instructions

To setup PerlTidy:

  1. Make sure you have Perl and PerlTidy installed. For Windows, grab the ActivePerl distribution from ActiveState and then download and install the PerlTidy script from SourceForge.
  2. In your toolbox tab, click the Add Item button at the top; then select New Command.
  3. The cursor starts off in the Command line, which you can specify as:

    perl c:\path\to\perltidy -b [perltidy style prefs] %f

    (-b specifies to tidy the file in place, making a .bak backup file.)

  4. In the advanced section, put %D in the Start field to launch PerlTidy in the script's working directory.
  5. Rename the Command to PerlTidy in the field at the top.
  6. At this point, you can also go to the Key Binding tab and assign a key combination to run the command.
  7. Save, open a Perl file and give it a shot, either using the key combination you assigned or by double clicking on the command from your Toolbox.

To setup a snippet, such as the HTML::Template tag:

  1. In your toolbox tab, click the Add Item button at the top, and then select New Snippet.
  2. Type in the text to insert when you run the snippet. In my case, I used:

    <!-- TMPL_VAR NAME="" ESCAPE=HTML -->
  3. Be sure to place the cursor between the quote marks and select the Maintain selected text or cursor position after insertion checkbox.
  4. Name the snippet (I used tmplvar) at the top. You can also go to the Key Binding tab at the top and assign a key combination to run the snippet.
  5. Save and then give it a shot, either using the key combination you assigned or by double clicking on the command from your Toolbox.
Visit the home of the Perl programming language: Perl.org

Sponsored by

Monthly Archives

Powered by Movable Type 5.13-en