December 2004 Archives

This Fortnight in Perl 6, December 7-20 2004

All~

The observant among you might notice that I missed last week's summary. With the hubbub and confusion of the holidays, I blame ninjas, in particular Ryu Hyabusa. Given that Christmas is next weekend and New Years is the weekend after that, what you are like to see in the future are a pair of ten-day summaries or some other equally irregular pattern. If you are thinking of using the dates of my summaries to seed a random number generator, I would advise against it as I can be really easily bought ;-). Without more ado, I give you this fortnight's summary starting with:

Perl 6 Language

Lexical Scope of Parametric Declaration Blocks

Ashley Winters wanted to know the differences between type parameter lists and sub parameter lists. Luke Palmer could not think of any.

Object Representation

Abhijit Mahabal noticed that S12 allowed one to supply an object layout to bless() and wondered if one could really have two instances of the same class with different layouts. Larry admitted that he had probably not intended for that to be the case.

Capturing into a Hash, Hypothetically

Patrick R. Michaud wondered about capturing things into a hash in S05, as <ident> now captures. Larry admitted that it was probably supposed to be (<<ident>>) but also noticed that this exposed a blind spot in the design. He ruminated about this blind spot and ways to solve it. After much churning, he decided that it would be possible to perform multiple different (but identically named) rule captures by adding information after a dash a la <ws-1> <ws-2> <ws-3>.

Custom Subscripting

When talking about key Type for a hash, Larry offhandedly commented about attaching a block to a hash or array to provided custom subscripting. Many people drooled over the awesome syntactic sugar this could provide them.

Undeclared Attributes

Dave Whipp hoped that he need not predeclare his attributes; as they necessarily start with $. The fact that a new variable is an attribute is easy to determine. Abhijit Mahabal thought that it would not be a good idea but then asked if classes could be declared as not strict. We're still waiting for more official word...

Classes Which Autovivify Attributes

Abhijit Mahabal wondered about creating a class that populates its attributes on demand, as some of them might be rarely used. Larry suggested that it would be something that one should not undertake lightly, and a simple hash attribute would provide most of what he wanted. This also morphed into the eternal debate about strictures and oneliners. There has to be a joke in there somewhere. A stricture, a oneliner, and Larry Wall walk into a bar....

Auto my

Rod Adams wondered if having my occur automatically for new variables might be worthwhile. Several people commented that some languages already do this, and it is simply an aesthetic choice. The consensus seems to be that Perl has already made this choice and will stick with its answer.

Perl 6 Compiler

At long last, Google has picked up P6C. I guess I have slightly mixed emotions about this, as it takes a running gag from me. Alas, I will have to find another.

PGE Tests

Markus Laire began working on a formerly small script to convert Perl 5's regex tests to PGE. He produced a modest 700 tests, a few of which pass. Nice work. Patrick suggested only running the script once and thereafter maintaining the tests external to Perl 5.

On your marks, get set, HACK!

Luke Palmer opened the door to hacking and has requested rules for parts of the Perl 6 Grammar. Patrick posted a link to its SVN repository.

Parrot

"\0namespace"

Leo committed a fix to support namespace mangling.

Store Global => Invalidate Method Cache

Leo committed a fix to invalidate the method cache when storing a global.

pow, hash, Batman Sound Effects!

Leo added pow and hash as vtables and opcodes. He also renamed new_extended to instantiate.

Base Scalar Semantics

Leo asked for comments about base PMC semantics and received none.

split Now Independent of Perl

James deBoer provided a patch removing the dependency on PerlArray in split. Will applied it.

SVN

Periodically, every project has a thread about switching some basic tool to another basic tool. This time, the thread is in P6I, and the tools in questions are CVS and SVN. Many voiced support, but no one decided anything with any permanence.

Continuations

At long last, the long-running continuation thread has died down. Unless I am mistaken, the status quo remains, and return continuations should restore register contents even when promoted and repeatedly invoked.

Correctly Dispatching Opcodes and Functions

Sam Ruby had some concern about dynamically overloading __add__ in Python. Leo and Sam had quite a bit of back and forth about the proper way to handle it. I am not sure what resolution they reached other than that it should work.

Dynamic libs and Tcl Issues

Klaas-Jan Stol had some problems with Tcl and dynamic libs. Sam Ruby provided the necessary fix for him.

Class Refactoring

Leo began refactoring base PMCs such as Integer. He also started a TODO of what remains. This somehow morphed into discussion of logical xor...

Register Coloring Issues

Dan posted a failing test case for the register coloring. Leo fixed it.

split on String vs Regex

James deBoer wondered about the split opcode's current insistence on a string. Some advocated making it a method on a class, while others wanted to change the opcode to take a PMC instead of a string.

self vs P2

Sam Ruby discovered that usages of P2 had broken, as it was no longer the object of a method. Fortunately, Leo provided the helpful self.

Linux PPC

Long ago, there were troubles with Linux on PPC. Recently, chromatic submitted a patch. More recently, Warnock applied.

Whether vs Weather

This morning it was cold and snowy. Sadly, instead of fixing my dreary weather, chromatic fixed a mistyped "whether."

Benchmarks as Test

Justin DeVuyst supplied a patch to use the benchmarks as tests. Leo applied it.

make [install|docs|monkeys]

Adrian Lambeck suggested a few improvements to the current make set up, including a cool sounding make doc-install. Warnock applied.

Dan Still Alive

Dan sent an apology about the egregious amount of work he had and assured everyone that he was actively trying to get caught up. In the mean time, Cc him on things that need his personal attention, like your Christmas lists.

IMCC Parser No Longer Chokes on Empty Sub

Will noted that gremlins had fixed a problem in IMCC with empty subs. Yay, gremlins!

Current Object Invocation

Dan discovered that Parrot put the current object into its place after calling invoke. This was, of course, bad, so Leo fixed it.

Configure Help

Somebody had trouble with Configure.pl. Leo pointed out the usefulness of --verbose=2.

./parrot nonexistant.pbc => core dump

Dan noticed that Parrot would core dump when given a nonexistance bytecode file. Matthew Zimmerman and chromatic fought to get a patch in first. One succeeded.

Python Dynclasses Build Issues

Will noticed a multiple definition problem in the py* dynclasses. Sam fixed it.

MMD Dispatch Problem

Jens Rieks reported a problem with mmd_dispatch_v_pnp. Leo could not reproduce it.

mod_parrot with Mandelbrot

Jeff Horwitz posted a link to a webpage using mod_parrot to generate ASCII Mandelbrot sets. Really cool, but I prefer Julia sets.

Documentation Shortcomings

Dave Brondsema pointed out that the main FAQ should include info on the IRC channels/hosts that Parroters use. He also noted that some PDDs were not available on the website. Warnock applies.

Scope Cleanup Issues

Leo and Dan hashed out some of the issues with scope clean up and stale registers keeping things alive.

WinXP Duild Issues

Nicu Ionita reported a problem with the build on WinXP. Leo fixed it.

runops_fromc

Sam Ruby provided a patch allowing runops_fromc access to registers. Leo applied it.

Parameter Fillin Problem

Dan posted about a problem he was having. Unforunately, he cannot make a simple test case against CVS head, and Leo can't reproduce it.

Class Autoload

Leo added support for autoloading dynamic classes for Python and Tcl. Sam Ruby suggested ways to extend it further.

P5 Is the New P2

Sam and Leo came to the conclusion that the current object should be passed in P5 as well as P2. Currently, they are waiting for Dan... nudge, nudge...

get_anonymous_subclass

Leo wondered what get_anonymous_subclass was for. Dan explained.

Context, Wrappers, Rules, NCI

Sam and Leo had a discussion about what exactly VTABLE functions, MMD functions, and ops should do. I think they spent much of the time talking past each other and all of it talking past me.

Namespaces As Objects

Sam Ruby wants to use namespaces as objects. Leo is not so sure that this is right. This turned into a dialog about find_method.

POD Cleanup

chromatic provided and threatened to apply a patch fixing up some of the POD nits.

Plain Ole Hash

Bernhard Schmalhofer provided a patch cleaning up Hash. Sam wondered if the NCI calls in Hash::fromkeys could go away along with a few others. Leo said that iterators in general needed another round of thought.

Opcode/Sub Conflict

Dave Brondsema notice that there was a problem with subs whose names conflict with opcodes. Luke Palmer provided a workaround.

Duplicate Subclass Naming Errors

Simon Glover noticed some problems with duplicate subclass names, notably that one could not create two nameless classes.

MMD and VTABLE_find

Leo suggested a mechanism for MMD and VTABLE_find. Sam Ruby provided some input.

Auxiliary Variables

Tomas Necas wondered about the necessity of auxiliary variables in Perl 6. Luke Palmer and Dan provided some answers.

N Register Stomping

Dan noticed that something stomped his N registers occasionally.

The Usual Footer

If you find these summaries useful or enjoyable, please consider contributing to the Perl Foundation to help support the development of Perl. You might also like to send feedback to ubermatt@gmail.com

Building a 3D Engine in Perl, Part 2


This article is the second in a series aimed at building a full 3D engine in Perl. The first article, Building a 3D Engine in Perl, covered basic program structure, opening an OpenGL window using SDL, basic projection and viewing setup, simple object rendering, object transformations, and depth ordering using the OpenGL depth buffer.

Editor's note: see also the rest of the series, lighting and movement, and profiling your application.

This time, I'll discuss rotating and animating the view, SDL event and keyboard handling, and compensating for frame rate variations. As a bonus, I'll demonstrate some real-world refactoring, including a conversion from procedural to (weakly) object-oriented code. Before I start, however, there were a couple of issues discovered since the first article went live:

  • In the first article, I wrote "orthogonal projection." This should be "orthographic projection," which reminds me again that no matter how many times you proofread, you can still miss the bug--in code or in prose. Unfortunately, prose is a bit harder to write tests for.
  • Todd Ross discovered a problem with SDL on FreeBSD 5.3, which caused the code to die immediately with "Bad system call (core dumped)." A short while later, he reported the workaround. He set his LD_PRELOAD environment variable with a little magic, and everything worked fine:

    setenv LD_PRELOAD /usr/lib/libc_r.so

    His research method is a good one to follow if you should encounter a similar problem. He installed another SDL_Perl application, in this case Frozen Bubble. Once he was sure it worked, he checked the code in the launcher script and found the magic shown above. A quick test confirmed that this worked for his code as well.

    Frozen Bubble is a 2D application, so if it works fine but your OpenGL programs don't, check to make sure that OpenGL works at all. Unix variants should supply the glxinfo and glxgears programs. Use glxinfo to gather details on your OpenGL driver; it serves as both a sanity check and a good addition to bug reports. glxgears does a simple animated rendering of meshing gears. This tells you whether OpenGL works correctly (at least for basic stuff) and what performance your OpenGL driver and hardware can provide. Both programs work under X on Apple's OS X 10.3 as well.

Keep your comments, questions, and bug reports coming! I'd like to recognize your contribution in the next article, but if you'd rather remain anonymous, that's fine too.

Without further ado, let's start. If you want try the code at each stage without all the typing, download the example source code. It includes a README.steps file that should help you follow along more easily.

Moving the Viewpoint

At the end of the last article, our scene had a set of axis lines roughly in the center of the screen, with a big white cube behind them and a rotated flat yellow box to the right:

sub draw_view
{
    draw_axes();

    glColor(1, 1, 1);
    glPushMatrix;
    glTranslate( 0, 0, -4);
    glScale( 2, 2, 2);
    draw_cube();
    glPopMatrix;

    glColor(1, 1, 0);
    glPushMatrix;
    glTranslate( 4, 0, 0);
    glRotate( 40, 0, 0, 1);
    glScale(.2, 1, 2);
    draw_cube();
    glPopMatrix;
}

Let's move that white cube to the right by changing the first glTranslate call as follows:

glTranslate( 12, 0, -4);

Now the right side of the window cuts off the white box. If I wanted to fix that while maintaining the relative positions of all the objects, there are a few possible changes I could make:

  • Use a wider projection (FOV) angle to see more of the scene at once. Unfortunately, it's already at 90 degrees, which is fairly wide. The perspective effect is already very strong; much wider, and the rendering will look too distorted.
  • Individually move all objects in the scene the same distance to the left. This would certainly work, but is a lot of effort, especially when there are many objects in the scene.
  • Move the viewpoint right to recenter the view. This is my preference.

I want to move the viewpoint to the right, a positive X direction, so I add +6 to the X component of the viewing translation:

sub set_view_3d
{
    glTranslate(6, -2, -10);
}

Now the scene is even farther to the right. The problem is that OpenGL combines the transformations used to modify the view (viewing transformations) with those used to transform objects in the scene (modeling transformations) in the modelview matrix. OpenGL has no way to know whether I intend any given modelview transformation to alter the view or the objects in the scene; it treats all of them as altering the objects. By translating +6 X, I effectively moved every object 6 units to the right, rather than moving my viewpoint right as intended.

I hinted at the solution before: moving the viewpoint right is equivalent to moving all objects in the scene to the left. The solution to this problem is to reverse the sign of the translation:

sub set_view_3d
{
    glTranslate(-6, -2, -10);
}

This puts the viewpoint at (6, 2, 10) where I wanted it, roughly recentering the scene. Now you can see why the viewing translation from the first article moved the viewpoint to a point slightly above (+Y) and some distance closer to the user (+Z) than the origin. I simply reversed the signs of the viewpoint coordinates I wanted, (0, 2, 10).

The scene is now centered, but with this static view, it's difficult to tell the true location and relative sizes of the objects in the scene. Perhaps I can rotate the view a bit to see this. I'll rotate it 90 degrees counterclockwise (positive rotation) around the Y axis:

sub set_view_3d
{
    glTranslate(-6, -2, -10);
    glRotate(90, 0, 1, 0);
}

Well, that certainly rotated things, but it's still hard to see where the objects really are. Why did the scene end up all over on the left like that and with the axis lines in front?

Animating the View

To understand what's really going on with an odd transformation, it helps me to turn it into a short animation. I start the animation with a very small transformation and keep increasing it until well past the intended level. This way, I can see the effect of both smaller and larger changes.

To do this, I need a few more frames in the animation. I can do this by changing the last line in draw_frame:

$done = 1 if $frame == 10;

I also want the rotation to animate with each frame:

sub set_view_3d
{
    glTranslate(-6, -2, -10);
    glRotate(18 * $frame, 0, 1, 0);
}

This chops the rotation into 18 degree increments, starting at frame 1 with an 18 degree rotation and ending at frame 10 with a 180 degree rotation.

Running this program shows what is happening. The scene rotates counterclockwise around its origin, the intersection point of the axis lines. I wanted to rotate the viewpoint, but I rotated the objects instead. Just reversing the sign won't do the trick. That will rotate the scene the other direction (clockwise), but it won't rotate around the viewpoint--it will still rotate around the scene origin.

In the first article, I described how to visualize a series of transforms by thinking about transforming the local coordinate system of the objects in a series of steps. Looking at the code above, you can see that it first translates the scene origin away and then rotates around that new origin. To rotate around the viewpoint, I need to rotate first and then translate the scene away:

sub set_view_3d
{
    glRotate(18 * $frame, 0, 1, 0);
    glTranslate(-6, -2, -10);
}

This now rotates around the viewpoint, but because it rotates 180 degrees starting from dead ahead, the scene ends up behind the viewpoint. To start the view so that the scene is on one side and then rotates to be on the other, I simply offset the angle:

sub set_view_3d
{
    glRotate(-90 + 18 * $frame, 0, 1, 0);
    glTranslate(-6, -2, -10);
}

At frame 1, the rotation angle is -90 + 18 * 1 = -72 degrees. At frame 10, the angle is -90 + 18 * 10 = 90 degrees. Perfect.

Stop and Turn Around

There's only one little problem: it's going the wrong way! I wanted to do a counterclockwise rotation of the view, but that should make the scene appear to rotate clockwise around the viewpoint. Imagine standing in front of a landmark, taking a picture. Looking through the viewfinder, you might notice that the landmark is a bit left of center. To center it, turn slightly left (counterclockwise as seen from above, or around +Y in the default OpenGL coordinate system). This would make the landmark appear to move clockwise around you (again as seen from above), moving it from the left side of the viewfinder to the center.

In this case, reverse the angle's sign:

sub set_view_3d
{
    glRotate(90 - 18 * $frame, 0, 1, 0);
    glTranslate(-6, -2, -10);
}

In fact, every transformation of the view is equivalent to the opposite transformation of every object in the scene. You must reverse the sign of the coordinates in a translation, reverse the sign of the angle in a rotation, or take the inverse of the factors in a scaling (shrinking the viewer to half size makes everything appear twice as big). As we saw before, you must reverse the order of rotation and translation as well.

Scaling is a special case. Inverting the factors works, but you must still do the scaling after the translation to achieve the expected effect, rather than following the rule for rotation and translation and reversing the transformation order completely. The reason is that scaling before the translation scales the translation also. Scaling by (2, 2, 2) would double the size of all of the objects in the scene, but it would also put them twice as far away, making them appear the same size. I'll skip the code for this and leave it as an exercise for the reader. Go ahead, have fun.

If you decide to give view scaling a try, remember that all distances will change. This affects some non-obvious things such as the third and fourth arguments to gluPerspective (the distance to the nearest and farthest objects OpenGL will render).

Smoothing It Out

After watching these animations for a while, the jerkiness really begins to bother me, and because I doubled the number of animation frames, it takes twice as long to finish a run. Both of these problems relate to the second-long sleep at the end of draw_frame. I should be able to fix them by shortening the sleep to half a second:

sleep .5;

Chances are, that doesn't yield quite the respected result. On my system, there's a blur for a fraction of a second, and the whole run is done. Unfortunately, the builtin Perl sleep function only handles integer seconds, so .5 truncates to 0 and the sleep returns almost instantly.

Luckily, SDL provides a millisecond-resolution delay function, SDL::Delay. To use it, I add another subroutine to handle the delay, translating between seconds and milliseconds:

sub delay
{
    my $seconds = shift;

    SDL::Delay($seconds * 1000);
}

Now, changing the sleep call to delay fixes it:

delay(.5);

The movement is faster and it only takes five seconds to complete the entire animation again, but this code still wastes the available performance of the system. I want the animation to be as smooth as the system allows, while keeping the rotation speed (and total time) constant. To implement this, I need to give the code a sense of time. First, I add another global to keep the current time:

my ($time);

At this point, my editor likely just sprayed his screen with whatever he's drinking and coughed "Another global?!?" I'll address that later in this article during the refactoring.

To update the time, I need a couple more functions:

sub update_time
{
    $time = now();
}

sub now
{
    return SDL::GetTicks() / 1000;
}

now calls SDL's GetTicks function, which returns the time since SDL initialization in milliseconds. It converts the result back to seconds for convenience elsewhere. update_time uses now to keep the global $time up to date.

main_loop uses this to update the time before rendering the frame:

sub main_loop
{
    while (not $done) {
        $frame++;
        update_time();
        do_frame();
    }
}

Because this version won't artificially slow the animation, I make two changes to draw_frame. I remove the delay call and change the animation end test to check whether the time has reached five seconds, instead of whether frame ten has been drawn.

sub draw_frame
{
    set_projection_3d();
    set_view_3d();
    draw_view();

    print '.';
    $done = 1 if $time >= 5;
}

Finally, set_view_3d must base its animation on the current time instead of the current frame. Our current rotation speed is 18 degrees per frame. With 2 frames per second, that comes to 36 degrees per second:

sub set_view_3d
{
    glRotate(90 - 36 * $time, 0, 1, 0);
    glTranslate(-6, -2, -10);
}

This version should appear much smoother. On my system, the dots printed for each frame scroll up the terminal window. If you run this program multiple times, you'll notice the number of frames (and hence dots) varies. Small variations in timing from numerous sources cause a frame now and then to take more or less time. Over the course of a run, this adds up to being able to complete a few frames more or less before hitting the five second deadline. Visually, the rotation speed should appear nearly constant because it calculates the current angle from the current time, whatever that may be, rather than the frame number.

Refactoring for Fun and Clarity

Now that the animation is smooth, I'm almost ready to add some manual control using SDL events. That's a big topic and involves a lot of code. It's always a good idea before a big change to step back, take a look at the exisiting code, and see if it needs a clean up.

The basic procedure is as follows:

  • Find one obvious bit of ugliness.
  • Make a small atomic change to clean it up.
  • Test to make sure everything still works.
  • Lather, rinse, repeat until satisfied.

Unfortunately, it's occasionally necessary to make one part of the code a little uglier while cleaning up something else. The trick is eventually to clean up the freshly uglified piece.

Refactoring the View

And on that note, let's add another global! I don't like the hardcoding of set_view_3d. I'd like to convert that into a data structure of some kind, so I define a view object:

my ($view);

This needs an update routine, so here's a simple one:

sub update_view
{
    $view = {
        position    => [6, 2, 10],
        orientation => [-90 + 36 * $time, 0, 1, 0],
    };
}

This is simply the position and orientation of the virtual viewer (before the sign reversal needed by the viewing transformations). I need to call this in the main loop, just before calling do_frame:

sub main_loop
{
    while (not $done) {
        $frame++;
        update_time();
        update_view();
        do_frame();
    }
}

At this point, running the program should show that nothing much changed because I haven't actually changed the viewing code--the new code runs in parallel with the old code. Using the new code requires replacement of set_view_3d:

sub set_view_3d
{
    my ($angle, @axis) = @{$view->{orientation}};
    my ($x, $y, $z)    = @{$view->{position}};

    glRotate(-$angle, @axis);
    glTranslate(-$x, -$y, -$z);
}

Running the program should again show that nothing visually changed, indicating a successful refactoring. At this point, you may wonder what this has gained; there's a new global and a dozen or so more lines of code. The new code has several benefits:

  • The concepts of animating the view parameters and setting the current view in OpenGL are now separate, as they should be.
  • Hoisting the update_view call up to main_loop next to the update_time call begins to collect all updates together, cleaning up the overall design.
  • The new code hints at further refactoring opportunities.

In fact, I can see several problem places to refactor next, along with some reasons to fix them:

  1. The mess of globals, since I just added one.
  2. Updating $done in draw_view (mixing updates and OpenGL work again) to continue collecting all updates together.
  3. Pervasive hardcoding in draw_view, for all the same reasons I refactored set_view_3d.

The Great Global Smashing

The globals situation is out of hand, so now seems a good time to fix that and check off the first item of the new "pending refactoring" list. First, I need to decide how to address the problem.

Here's the current globals list:

my ($done, $frame);
my ($conf, $sdl_app);
my ($time);
my ($view);

In this mess, I see several different concepts:

  • Configuration: $conf
  • Resource object: $sdl_app
  • Engine state: $done, $frame
  • Simulated world: $time, $view

I'll create from these groupings a single engine object laid out like so (variables show where the data from the old globals goes):

{
    conf     => $conf,
    resource => {
        sdl_app => $sdl_app,
    },
    state    => {
        done    => $done,
        frame   => $frame,
    },
    world    => {
        time    => $time,
        view    => $view,
    }
}

This is a fairly radical change from the current code, so to be safe, I'll do it in several small pieces, testing after each version that everything still works.

The first step is to add a constructor for my object:

sub new
{
    my $class = shift;
    my $self  = bless {}, $class;

    return $self;
}

This is pretty much the garden variety trivial constructor in Perl 5. It blesses a hash reference into the specified class and then returns it. I also need to change my START code to use this new constructor to create an object and call main as a method on it:

START: __PACKAGE__->new->main;

This snippet constructs a new object, using the current package as the class name, and immediately calls the main method on the returned object. main doesn't have any parameters, so calling it as a method won't affect it (there are no existing parameters that would be shifted right by a new invocant parameter). I never store the object in a variable as a form of self-imposed stricture. Because the object only exists as the invocant of main, I must convert every routine that accesses the information in the object to a method and change all calls to those routines as well.

Let It Flow

Testing at this point shows all still works, so the next change is to make main flow the object reference through to its children by calling them as methods:

sub main
{
    my $self = shift;

    $self->init;
    $self->main_loop;
    $self->cleanup;
}

Testing this shows that, again, all is fine, as expected. init and main_loop are changed in the obvious fashion (cleanup doesn't do much, so it doesn't need to change now):

sub init
{
    my $self = shift;

    $| = 1;

    $self->init_conf;
    $self->init_window;
}

sub main_loop
{
    my $self = shift;

    while (not $done) {
        $frame++;
        $self->update_time;
        $self->update_view;
        $self->do_frame;
    }
}

Notice that I have not changed the references to $done and $frame in main. It's important to make only a single conceptual change at a time during refactoring, to minimize the chance of making an error and not being able to figure out which change caused the problem. I'll return to these references in a bit. Testing this version shows that all is well, so I continue:

sub do_frame
{
    my $self = shift;

    $self->prep_frame;
    $self->draw_frame;
    $self->end_frame;
}

sub draw_frame
{
    my $self = shift;

    $self->set_projection_3d;
    $self->set_view_3d;
    $self->draw_view;

    print '.';
    $done = 1 if $time >= 5;
}

Again, for this pass, I ignore $done and $time in draw_frame. At this point, I've pretty much exhausted all of the changes that amount to simply turning subroutine calls into method calls and the code still works as advertised.

Replacing the Globals

With this working, I start into more interesting territory. It's time to move the globals into their proper place in the object. First up are the state variables $done and $frame in main_loop:

sub main_loop
{
    my $self = shift;

    while (not $self->{state}{done}) {
        $self->{state}{frame}++;
        $self->update_time;
        $self->update_view;
        $self->do_frame;
    }
}

and the last line of draw_frame:

$self->{state}{done} = 1 if $time >= 5;

As they are no longer globals, I remove their declarations as well. I will have to come back to draw_frame again when cleaning up $time. One change per pass--it's very easy to follow a long chain of related changes before doing a test run and find out you made a mistake. Somewhere. Argh. In this case, I resist the urge to keep changing the code and do another test run immediately to find that indeed all still works.

Next up is the world attribute $view:

sub update_view
{
    my $self = shift;

    $self->{world}{view} = {
        position    => [6, 2, 10],
        orientation => [-90 + 36 * $time, 0, 1, 0],
    };
}

sub set_view_3d
{
    my $self = shift;

    my $view           = $self->{world}{view};
    my ($angle, @axis) = @{$view->{orientation}};
    my ($x, $y, $z)    = @{$view->{position}};

    glRotate(-$angle, @axis);
    glTranslate(-$x, -$y, -$z);
}

In set_view_3d, it seemed clearest to make a lexical $view loaded from the object. This allowed me to leave the rest of the function clean and unchanged. Testing after the above changes and removing the global declaration for $view shows that all is good.

Next up are $conf and the resource object $sdl_app, following much the same pattern as before:

sub init_conf
{
    my $self = shift;

    $self->{conf} = {
        title  => 'Camel 3D',
        width  => 400,
        height => 400,
        fovy   => 90,
    };
}

sub init_window
{
    my $self = shift;

    my $title = $self->{conf}{title};
    my $w     = $self->{conf}{width};
    my $h     = $self->{conf}{height};

    $self->{resource}{sdl_app}
        = SDL::App->new(-title  => $title,
                        -width  => $w,
                        -height => $h,
                        -gl     => 1,
                       );
    SDL::ShowCursor(0);
}

sub set_projection_3d
{
    my $self   = shift;

    my $fovy   = $self->{conf}{fovy};
    my $w      = $self->{conf}{width};
    my $h      = $self->{conf}{height};
    my $aspect = $w / $h;

    glMatrixMode(GL_PROJECTION);
    glLoadIdentity;
    gluPerspective($fovy, $aspect, 1, 1000);

    glMatrixMode(GL_MODELVIEW);
    glLoadIdentity;
}

sub end_frame
{
    my $self = shift;

    $self->{resource}{sdl_app}->sync;
}

The first time I did this, it broke. I had forgotten to make the changes to set_projection_3d. Thanks to use strict, the error was obvious, and a quick fix later, everything worked again.

Last but not least, it's time to fix the remaining world attribute $time:

sub update_time
{
    my $self = shift;

    $self->{world}{time} = now();
}

In update_view, I continue with my tactic of creating lexicals and leaving the remaining code alone:

my $time = $self->{world}{time};

Finally, the last line of draw_frame changes again:

$self->{state}{done} = 1
    if $self->{world}{time} >= 5;

The first test run of the completed refactoring uncovered a typo that gave an obscure warning. Thankfully, I only had to check the few changed lines since the last test, and the typo was easily found. With things working again, The Great Global Smashing is complete. The once completely procedural program is now on its way to claiming object orientation. (Boy will I be happy to switch to Perl 6 OO syntax! Perl 6 OO keeps the visual clarity of pure procedural code while gaining several powerful benefits not available in Perl 5. I could fake the clearer syntax with judicious use of source filtering, but that's another article.)

This seems to me like enough refactoring for now, so it's back to the main thrust of development: keyboard control.

The Big Event

Keyboard handling is a special case of SDL event handling, and not an entirely trivial case at that. I'll start with the basic structure for processing SDL events and handle a much simpler event first. To access SDL events, I need to load the SDL::Event module:

use SDL::Event;

Like SDL::App, the code needs to keep track of an SDL::Event resource object to access the event queue. In addition, I need to keep track of which routine I'll use to process each event type. This is a new kind of data, so I add a new branch to the engine object for various lookup tables. To set up both of these, I add a new initialization function:

sub init_event_processing
{
    my $self = shift;

    $self->{resource}{sdl_event} = SDL::Event->new;
    $self->{lookup}{event_processor} = {
        &SDL_QUIT    => \&process_quit,
    };
}

SDL event types are constants in the general SDL constant convention (UPPERCASE with a leading SDL_ marker). The event type for quit events is SDL_QUIT, which I associate with the process_quit routine using a subroutine reference.

A new line at the end of init calls the initialization routine:

$self->init_event_processing;

Every time through, the main loop should process events before updating the view (after I add keyboard control, the view should update using the latest user input). The contents of the loop in main_loop are now as follows:

$self->{state}{frame}++;
$self->update_time;
$self->do_events;
$self->update_view;
$self->do_frame;

do_events is very simple at this stage, just calling process_events to, er, process pending SDL events:

sub do_events
{
    my $self = shift;

    my $queue = $self->process_events;
}

The Event Processing Loop

process_events is where all the magic happens:

sub process_events
{
    my $self = shift;

    my $event  = $self->{resource}{sdl_event};
    my $lookup = $self->{lookup}{event_processor};
    my ($process, $command, @queue);

    $event->pump;
    while (not $self->{state}{done} and $event->poll) {
        $process = $lookup->{$event->type} or next;
        $command = $self->$process($event);
        push @queue, $command if $command;
    }

    return \@queue;
}

The first couple of lines provide shorter names for the previously stored SDL::Event object and event processor lookup table. The rest of the variables respectively store:

  • A reference to the processing routine for the current event
  • The internal command to convert the event into
  • The queue of commands collected from incoming events

The core of the code starts by telling the SDL::Event object to gather any pending operating system events in preparation for the processing loop, using the pump method. The processing loop checks to make sure that a previous event has not flagged the done state, which helps to improve responsiveness to quit events. Assuming that this has not happened, the loop requests the next SDL event using SDL::Event::poll. poll returns a false value when there are no events ready for pickup, thereby exiting the loop.

The first line inside the loop uses the event type to look up the proper event processing routine. If there is none, I use next to loop again and check the next event. Otherwise, the next line calls the processing routine as a dynamically chosen method to handle the event. If the processing routine determines that the event requires additional work, it should return a command packet to be queued. If the event should be ignored, the processor should simply return a false value.

The last line within the loop adds the command packet (if any) to the queue awaiting further processing. Once the loop processes all available SDL events, process_events returns the queue so that do_events can perform the next stage of processing.

It may seem confusing that each time through the loop the code reuses the same $event. You might expect SDL::Event::poll to return the next waiting event (and perhaps undef when none remain). Instead, the SDL API specifies that poll copies the data from the next entry in the event queue into the event object, returning a true or false status indicating whether this operation succeeded. As with some of the OpenGL quirks, SDL_Perl copies this odd interface directly, easing the transition for programmers used to the C API.

A consequence of this interface decision is that the event processing routine must make a copy of any data from the SDL event object needed for later. The call to SDL::Event::poll in the next iteration of the processing loop will overwrite any data left in the SDL event object, so simply storing the object reference won't work.

The process_quit routine doesn't need to save any data; it only matters that an SDL_QUIT event occurred:

sub process_quit
{
    my $self = shift;

    $self->{state}{done} = 1;
    return 'quit';
}

process_quit first sets the done state flag, which causes the loop in process_events to exit early and, more importantly, exits main_loop. It returns the simplest type of command packet, a string indicating the quit command. At this point, there's no code to process this command further, but this keeps things parallel with the keyboard version I'll show next.

What does all this buy us? For starters, we can now (finally) quit the program using the window manager before the animation runs its course. On my system, that means clicking the 'X' on the window's title bar. Still, that's not the same as having a quit key (which I find much more convenient).

Key Binding

To add a quit key, I first need to decide which key should quit the program. I'd choose the Escape key because that makes mnemonic sense to me, but everyone has their favorite, so I'll allow that to be a configuration setting. To do this, I extend the configuration hash with a new bind section:

sub init_conf
{
    my $self = shift;

    $self->{conf} = {
        title  => 'Camel 3D',
        width  => 400,
        height => 400,
        fovy   => 90,
        bind   => {
            escape => 'quit',
        }
    };
}

Now anyone who wants to choose a different quit key can simply change the keyboard bindings hash. In fact, several keys could be associated with the same command, so that either the Escape key or 'q' would exit the program. The hash value corresponding to each specified key is the command packet issued when the user presses that key. This one matches the command packet I'd chosen for the window manager quit message earlier.

Next, I need to process keypress events, which have the event type SDL_KEYDOWN. I add another entry to the event_processor hash:

sub init_event_processing
{
    my $self = shift;

    $self->{resource}{sdl_event} = SDL::Event->new;
    $self->{lookup}{event_processor} = {
        &SDL_QUIT    => \&process_quit,
        &SDL_KEYDOWN => \&process_key,
    };
}

and define the key processor as follows:

sub process_key
{
    my $self = shift;

    my $event   = shift;
    my $symbol  = $event->key_sym;
    my $name    = SDL::GetKeyName($symbol);
    my $command = $self->{conf}{bind}{$name} || '';

    return $command;
}

process_key starts by extracting the key symbol from the SDL event. Key symbols are rather opaque for our purposes, so I request the key name matching the extracted key symbol using SDL::GetKeyName. This produces a friendly key name that I look up in the key bindings hash to find the appropriate command packet. If there is none, no matter; that key isn't bound yet so it yields an empty command packet. process_key then returns the command packet to add to the queue for further processing.

Handling Command Packets

At this point, the code converts a press of the Escape key into a quit command packet, but do_events ignores that packet because it does not process the command queue it receives from process_events. To make something happen, I first need to associate each known command with an action routine. I create a new lookup hash for this association, initialized in init_command_actions:

sub init_command_actions
{
    my $self = shift;

    $self->{lookup}{command_action} = {
        quit      => \&action_quit,
    };
}

As usual, I call this at the end of init:

$self->init_command_actions;

It's now time to fill out do_events:

sub do_events
{
    my $self   = shift;

    my $queue  = $self->process_events;
    my $lookup = $self->{lookup}{command_action};
    my ($command, $action);

    while (not $self->{state}{done} and @$queue) {
        $command = shift @$queue;
        $action  = $lookup->{$command} or next;
        $self->$action($command);
    }
}

This is similar in form to process_events. Instead of processing events from SDL's internal queue to create a queue of command packets, it processes queued command packets into actions to perform. The loop starts as usual by checking that the done is not true and that there are still commands pending in the queue.

Within the loop, it shifts the next command off the front of the queue. The next line determines the action routine associated with the command. If it cannot find one, it uses next to skip to the next command. Otherwise, it calls the action routine as a dynamically chosen method with the command packet as a parameter. This allows a single action routine to process several similar commands while still being able to tell the difference between them. I'll need this later for processing movement keys.

For all of that, action_quit is very simple; it just flags done:

sub action_quit
{
    my $self = shift;

    $self->{state}{done} = 1;
}

At this point, the Escape key really will quit the program early, and the window manager quit still works as well.

Now that the user can quit whenever desired, I can finally remove the incongruous end of draw_frame. It's no longer necessary to force the program to end after five seconds, and the dots printed each frame have outlived their usefulness. The routine now looks like this:

sub draw_frame
{
    my $self = shift;

    $self->set_projection_3d;
    $self->set_view_3d;
    $self->draw_view;
}

Now, if you wait long enough after the objects disappear on the right, the view rotates all the way around, and the scene appears again on the left. This version of the routine is much cleaner and incidently closes the next open refactoring issue (changing engine state within a drawing routine) for free.

Controlling the View

Now that the code can handle keypress events, it's time to control the view using the keyboard.

Instead of having the view completely recalculated every frame, I'd rather have each keypress modify the existing view state. To specify the initial state, I add another initialization routine:

sub init_view
{
    my $self = shift;

    $self->{world}{view} = {
        position    => [6, 2, 10],
        orientation => [0, 0, 1, 0],
        d_yaw       => 0,
    };
}

The new entry, d_yaw, tells update_view if there is a pending change (aka delta, hence the leading d_) in facing. The code so far can only handle yaw (left and right rotation), so that's the only delta key needed right now.

init calls this routine as usual in its new last line:

$self->init_view;

update_view applies the yaw delta to the view orientation, then zeroes out d_yaw so that it won't continue to affect the rotation in succeeding frames (without the user pressing the rotation keys again):

sub update_view
{
    my $self   = shift;

    my $view   = $self->{world}{view};

    $view->{orientation}[0] += $view->{d_yaw};
    $view->{d_yaw}           = 0;
}

A command action assigned to the yaw_left and yaw_right commands updates d_yaw:

sub init_command_actions
{
    my $self = shift;

    $self->{lookup}{command_action} = {
        quit      => \&action_quit,
        yaw_left  => \&action_move,
        yaw_right => \&action_move,
    };
}

To assign keys for these commands, I update the bind hash in init_conf:

bind   => {
    escape => 'quit',
    left   => 'yaw_left',
    right  => 'yaw_right',
}

The big change is the new command action routine action_move:

sub action_move
{
    my $self = shift;

    my $command     = shift;
    my $view        = $self->{world}{view};
    my $speed_yaw   = 10;
    my %move_update = (
        yaw_left  => [d_yaw =>  $speed_yaw],
        yaw_right => [d_yaw => -$speed_yaw],
    );
    my $update = $move_update{$command} or return;

    $view->{$update->[0]} += $update->[1];
}

action_move starts by grabbing the command parameter and current view. It then sets the basic rotation speed, measured in degrees per key press. Next, the %move_update hash defines the view update associated with each known command. If it knows the command, it retrieves the corresponding update. If not, action_move returns.

The last line interprets the update. The view key specified by the first element of the update array is incremented by the amount specified by the second element. In other words, receiving a yaw_left command causes the routine to add $speed_yaw to $view->{d_yaw}; a yaw_right command adds -$speed_yaw to $view->{d_yaw}, effectively turning the view the opposite direction.

With these changes in place, the program starts up looking directly at the scene as it appeared near the beginning of this article. Each press of the left or right arrow keys turns the view ten degrees in the appropriate direction (remember that the scene appears to turn the opposite direction around the view). Holding the keys down does nothing; only a change from unpressed to pressed does anything, and it only rotates the view one increment. This, as they say, is suboptimal.

Angular Velocity

In order to solve this, the code has to change from working purely in terms of angular position to working in terms of angular velocity. Pressing a key should start the view rotating at a constant speed, and it should stay that way until the key is released.

Velocity goes hand in hand with time. In particular, for each frame, update_view needs to know how much time has passed since the last frame to determine the change in angle matching the rotation speed. To compute this time delta, the first change is to make sure the code always has a valid world time by initializing it at program start:

sub init_time
{
    my $self             = shift;

    $self->{world}{time} = now();
}

Of course, this requires another line at the end of init:

$self->init_time;

With this in place, I can change update_time to record the time delta for each frame:

sub update_time
{
    my $self = shift;

    my $now  = now();

    $self->{world}{d_time} = $now - $self->{world}{time};
    $self->{world}{time}   = $now;
}

I've made a few changes that shouldn't affect the behavior of the program, and I'm about to make several more that definitely will change the behavior, so now is a good time for a quick sanity test. All is fine, so it's time to contemplate the design for the remaining code.

Continuing Commands

There are really two classes of keyboard commands that I want to handle:

  • Single-shots like quit, drop_object, and pull_pin
  • Continuing commands like yaw_left and scream_head_off

To differentiate them, I borrow an existing game convention and use a leading + to indicate a continuing command. This changes the bind mapping in init_conf:

bind   => {
    escape => 'quit',
    left   => '+yaw_left',
    right  => '+yaw_right',
}

and the command_action lookup:

sub init_command_actions
{
    my $self = shift;

    $self->{lookup}{command_action} = {
          quit       => \&action_quit,
        '+yaw_left'  => \&action_move,
        '+yaw_right' => \&action_move,
    };
}

To process the key release events, I need to assign an event processor for the SDL_KEYUP event. I'll reuse the existing process_key routine:

sub init_event_processing
{
    my $self = shift;

    $self->{resource}{sdl_event} = SDL::Event->new;
    $self->{lookup}{event_processor} = {
        &SDL_QUIT    => \&process_quit,
        &SDL_KEYUP   => \&process_key,
        &SDL_KEYDOWN => \&process_key,
    };
}

process_key needs some training to be able to differentiate the two types of events:

sub process_key
{
    my $self    = shift;

    my $event   = shift;
    my $symbol  = $event->key_sym;
    my $name    = SDL::GetKeyName($symbol);
    my $command = $self->{conf}{bind}{$name} || '';
    my $down    = $event->type == SDL_KEYDOWN;

    if ($command =~ /^\+/) {
        return [$command, $down];
    }
    else {
        return $down ? $command : '';
    }
}

The new code (everything after the my $command line) first sets $down to true if the key is being pressed or to false if the key is being released. The remaining changes replace the old return $command line. For continuing commands (those that start with a +), there's a new class of command packet, containing both the $command and the $down boolean to indicate whether the command should begin or end. Single-shot commands (those without a leading +), send a simple command packet only for keypresses; they ignore key releases.

To handle the new class of command packets, I update do_events as well:

sub do_events
{
    my $self   = shift;

    my $queue  = $self->process_events;
    my $lookup = $self->{lookup}{command_action};
    my ($command, $action);

    while (not $self->{state}{done} and @$queue) {
        my @args;
        $command          = shift @$queue;
        ($command, @args) = @$command if ref $command;

        $action = $lookup->{$command} or next;
        $self->$action($command, @args);
    }
}

The only new code is inside the loop. It starts off by assuming that the command packet is a simple one, with no arguments. If the command turns out to be a reference instead of a string, it unpacks it into a command string and some arguments. The $action lookup remains unchanged, but the last line changes slightly to add @args to the parameters of the action routine. If there are no arguments, this has no effect, so a single-shot action routine such as action_quit can remain unchanged.

View, Meet Velocity

The view needs to keep track of the current yaw velocity and the velocity delta when the user presses or releases a key; I initialize them to 0 in init_view:

sub init_view
{
    my $self = shift;

    $self->{world}{view} = {
        position    => [6, 2, 10],
        orientation => [0, 0, 1, 0],
        d_yaw       => 0,
        v_yaw       => 0,
        dv_yaw      => 0,
    };
}

update_view needs a few more lines to handle the new variables:

sub update_view
{
    my $self   = shift;

    my $view   = $self->{world}{view};
    my $d_time = $self->{world}{d_time};

    $view->{orientation}[0] += $view->{d_yaw};
    $view->{d_yaw}           = 0;

    $view->{v_yaw}          += $view->{dv_yaw};
    $view->{dv_yaw}          = 0;
    $view->{orientation}[0] += $view->{v_yaw} * $d_time;
}

After adding any velocity delta to the current yaw velocity, this method multiples the total yaw velocity by the time delta for this frame to determine the change in orientation. This is accumulated with the current orientation and any other facing change for this frame.

Finally, I update action_move to handle the new semantics:

sub action_move
{
    my $self = shift;

    my ($command, $down) = @_;
    my $sign             = $down ? 1 : -1;
    my $view             = $self->{world}{view};
    my $speed_yaw        = 36;
    my %move_update      = (
        '+yaw_left'  => [dv_yaw =>  $speed_yaw],
        '+yaw_right' => [dv_yaw => -$speed_yaw],
    );
    my $update = $move_update{$command} or return;

    $view->{$update->[0]} += $update->[1] * $sign;
}

The $sign variable converts the $down parameter from 1/0 to +1/-1. I changed the last line of the routine to multiply the delta by this sign before updating the value. Adding a negated value is the same as subtracting the original value; this means that pressing a key requires adding the update, and releasing it will subtract it back out.

To make sure the new yaw commands update velocity, I also fixed up the %move_update hash to update dv_yaw instead of d_yaw and used the + versions of the command names. Finally, to bring back the old rotation rate, I set $speed_yaw to 36 degrees per second.

This version responds the way most people expect. Holding down a key turns the proper direction until the key is released. What about when the user presses multiple keys at once? This is why I was careful always to accumulate updates and deltas by using += instead of plain old =. If the user holds both the right and left arrow keys down at the same time, the view remains motionless because they've added in equal and opposite values to dv_yaw. If the user releases just one of the keys, the view rotates in the proper direction for the key that is still held down because the opposing update has now been subtracted back out. Press the released key back down while still holding the other, and the rotation stops again as expected.

Of course, there's no requirement that the speeds for yawing left and right must be the same. In fact, for an airplane or spaceship simulation, the game engine might set these differently to represent damage to the control surfaces or maneuvering thrusters. It may even be part of the gameplay to hold both direction keys down at the same time to compensate partially for this damage, perhaps tapping one key while holding the other steady.

One thing that doesn't magically work is making sure that if several keys map to the same command, pressing them all won't make the command take effect several times over. As it stands, the user could map five keys to the same movement command and move five times as fast. You might try fixing this on your own as a quick puzzle; I'll try to address it in a later installment.

Eyes in the Back of Your Head

You might be curious why I left d_yaw hanging around, since nothing uses it now. I could use it in the above-mentioned space simulation to simulate a thruster stuck on--continuously trying to veer the ship off course. In a first-person game, it allows one of my favorite commands, +look_behind. Holding down the appropriate key rotates the view 180 degrees. Releasing the key snaps the view back forward. To implement this, I need to add another entry to the bind hash:

bind   => {
    escape => 'quit',
    left   => '+yaw_left',
    right  => '+yaw_right',
    tab    => '+look_behind',
}

Then another command_action entry:

sub init_command_actions
{
    my $self = shift;

    $self->{lookup}{command_action} = {
          quit         => \&action_quit,
        '+yaw_left'    => \&action_move,
        '+yaw_right'   => \&action_move,
        '+look_behind' => \&action_move,
    };
}

Last but not least, another entry in %move_update:

my %move_update      = (
    '+yaw_left'    => [dv_yaw =>  $speed_yaw],
    '+yaw_right'   => [dv_yaw => -$speed_yaw],
    '+look_behind' => [d_yaw  =>  180       ],
);

That's it: a whopping three lines, all of which were entries in lookup hashes.

Conclusion

That's it for this article; it's already quite long. I started where I left off in the last article. From there, I talked about translation and rotation of the view; millisecond resolution SDL time; animation from jerky beginnings to smooth movement; basic SDL event and keyboard handling; single-shot and continuing commands; and a whole lot of refactoring.

Next time, I'll talk about moving the position of the viewpoint, clean up draw_view, and spend some more time on the OpenGL side of things with the basics of lighting and materials. In the meantime, I've covered quite a lot in this article, so go forth and experiment!

Introducing mod_parrot

It's been almost nine years since the first release of mod_perl, and it remains a very powerful tool for writing web applications and extending the capabilities of the Apache web server. However, lurking around the corner is Perl 6, which gives us not only a new version of Perl to embed in Apache but an entirely new runtime engine called Parrot. If there is ever going to be a Perl 6 version of mod_perl, Apache must first be able to run Parrot bytecode. This article introduces mod_parrot, an Apache module that allows the execution of Parrot bytecode from within the web server. Like mod_perl, it also gives your code direct access to the Apache API so you can write your own handlers.

What is Parrot?

Parrot is a virtual machine (VM) optimized for dynamic languages like Perl, Python, PHP, and Ruby. Source code written in each of these languages eventually compiles down to bytecode (after some optimizations), which subsequently runs in a virtual machine. Currently, each language runs bytecode with its own VM, but one of Parrot's goals is to provide a single common VM for all dynamic languages. This makes implementing a new language much easier because there's no need to worry about writing a new VM, and this also makes it possible for code in one language to call code or access data structures from another language.

Parrot code comes in three distinct flavors:

  • Bytecode: This is the file format natively interpreted by Parrot.
  • PASM: Parrot assembler (PASM) is the low-level language that compiles down to bytecode. It has very simple operations to perform functions such as setting registers, adding numbers, and printing strings. PASM is very straightforward, but it operates at such a low level that it can be quite cumbersome.
  • PIR: Parrot Intermediate Representation (PIR) solves many of the problems encountered when programming in PASM. It provides more user-friendly and compiler-friendly constructs and optimizations and feels more like a traditional high-level programming language. Parrot eventually breaks down PIR into PASM before compiling to bytecode (you can even include PASM blocks in PIR). All of the examples in this article use PIR.

For more information on Parrot, including PASM and PIR syntax, visit the Parrot website. It will provide a good background for understanding the code in this article.

Why mod_parrot?

Before discussing the details, you should know a little about mod_parrot's history. Ask Björn Hansen and Robert Spier originally wrote mod_parrot in 2002, later turning it over to Kevin Falcone. This version of mod_parrot targeted Apache 1.3 and had very limited functionality due to Parrot's immaturity at the time. In August 2004, with Parrot and its API much more mature, people suggested that the development on mod_parrot continue. This is where I picked up the project. However, instead of picking up where Ask, Robert, and Kevin left off, I started from scratch, coding for Apache 2 and focusing on access to the Apache API.

The new mod_parrot project has three primary goals:

  • Provide access to the Apache API through Parrot objects
  • Provide a common Apache layer for Parrot-based languages
  • Support for new languages should require little or no C coding
  • .

Let's discuss each of these in more detail.

Provide Access to the Apache API Through Parrot

Much of mod_perl's power comes from direct access to the Apache API. Rather than restrict your code to content generation, mod_perl provides hooks for things such as authentication handlers and output filters and gives you access to Apache's internal structures, all in Perl. Once you have this functionality, it is easy to implement other useful features including script caching and persistent database connections.

mod_parrot shares this approach, providing access to the Apache API from Parrot. It does this using Parrot objects, mimicking mod_perl's use of $r. There will eventually be hooks for all phases of the Apache lifecycle, though the current version supports only content handlers and authentication handlers.

Provide a Common Apache Layer for Parrot-based Languages

There are several different languages that can run inside Apache today. The major players here are mod_perl and PHP, but Python, Ruby, and even LISP have modules embedding them into Apache. Each of these implementations comes with its own Apache module, which makes sense for languages with different runtime engines. This is where Parrot changes the landscape dramatically-all languages targeted to the Parrot VM now have a common runtime engine, so they need only one Apache module: mod_parrot.

Support for New Languages Should Require Little or No C Coding

mod_parrot will provide all of the infrastructure for accessing the Apache API. The actual Apache module will already be written. Hooks for calling Parrot code for each stage of the Apache lifecycle will exist. Parrot objects will provide access to the Apache API. With all of this already done, and assuming our language compiles down to Parrot bytecode, we should be able to write the "glue" between Apache and our language in the language itself. mod_perl could be written in Perl; mod_python could be written in Python, and so on. Very little C code, if any, would be necessary. Each language community could maintain its own language-specific code while sharing the mod_parrot module.

Architecture

mod_parrot is written for Apache 2, with no plans to back-port it to Apache 1.3. The reason behind this decision is to code for the future, not the past or present; after all, Perl 6 is still a few years down the road. It's also much easier to write a module for Apache 2 than it is for 1.3! In addition to the Apache 2 decision, there are several other interesting aspects of the mod_parrot architecture.

NCI

The most significant design decision is the use of NCI (native call interface) to access the Apache API. mod_perl accesses most of the Apache API functions through individual XS wrappers (basically a bunch of C macros), themselves compiled into mod_perl itself or its supporting modules. This is a tried and true method, used for many Perl modules as well. Now, Parrot gives us NCI, which eliminates the need for these wrappers, letting you call arbitrary C functions without having to write any C code. Here's an example of a Parrot program that calls the C function getpid(), which returns the current process ID:

.sub _main
    # load libc.so, where getpid() is defined, and assign it to $P0
    $P0 = loadlib '/lib/libc.so.6'

    # find the function in the library and assign it to $P1
    # 'iv' means that getpid() returns an integer and takes no arguments
    $P1 = dlfunc $P0, 'getpid', 'iv'

    # call getpid() and place result in $I0
    $I0 = $P1( )

    # print the PID
    print $I0
    print "\n"
.end

That's it--there is no C code to write, no recompilation, and no relinking. However, the Apache API functions do not come from a loadable shared library; they're in the Apache executable, httpd. Fortunately, NCI can run C functions contained in the running process image, solving that problem. For more information on NCI, see the Parrot NCI Documentation.

The Apache::RequestRec Object

All access to the Apache API goes through Parrot objects. Because mod_parrot borrows heavily from mod_perl, it made sense to base the primary object class in mod_parrot on Apache's request_rec structure. Just as in mod_perl, the class is Apache::RequestRec. This name is subject to change, however, as Parrot's namespace nomenclature becomes clearer.

Every method is written in Parrot, with NCI calls to their corresponding Apache API functions. For example, here is the Parrot method for the ap_rputs function ($r->puts in mod_perl):

.sub puts method, prototyped
    .param string data
    .local pmc r
    .local pmc ap_rputs
    .local int offset

    classoffset offset, self, 'Apache::RequestRec'
    getattribute r, self, offset

    # find NCI object for ap_rputs
    find_global ap_rputs, 'Apache::NCI', 'ap_rputs'

    # use NCI to call out to Apache's ap_rputs
    ap_rputs( data, r )
.end

Currently, Apache::RequestRec is the only class implemented in mod_parrot. Other classes to support the API will eventually appear, including classes to support Apache's conn_rec and server_rec structures.

Installing mod_parrot

You can download mod_parrot from the mod_parrot home page. Additionally, you'll need the following prerequisites (as of version 0.1):

  • Parrot 0.1.1 (Poicephalus)
  • Apache 2.0.50 or later
  • Perl 5.6.0 or later (for configuration only)
  • Apache::Test 1.13 or later (for the test suite)

Once you have all the prerequisite software, run the Configure.pl script. The arguments to this script will most certainly change in future releases, but for now, there are only two arguments:

  • --parrot-build-dir=path-to-parrot-source, the path to the top-level Parrot directory
  • --apxs=path-to-apxs, the path to Apache's apxs script, usually found in the bin directory under the Apache installation directory
$ perl Configure.pl --parrot-build-dir=../parrot \
  --apxs=/usr/local/apache2/bin/apxs

Generating Makefile...done.
Creating testing infrastructure...done.

Type 'make' to build mod_parrot.

When configuration completes, type make to build mod_parrot, then make test to run the tests. To install, become root and type make install. This will install the mod_parrot module into your Apache installation and activate the module in httpd.conf.

Writing a mod_parrot Handler

While there are currently no languages targeted to Parrot that have the object support to use mod_parrot (though Parakeet in the Parrot source looks promising), we can still write Apache handlers. In what language? In Parrot, of course! Well, actually, PIR. Here's a simple content handler that displays "Hello World," or if you pass it an query string in the URL, "Hello name," where name is the string you pass.

# this namespace is used to identify the handler
.namespace [ 'HelloWorld' ]

# the actual handler
.sub _handler
    # our Apache::RequestRec object
    .local pmc r

    # this will contain Apache constants
    .local pmc ap_constants

    # instantiate the Apache::RequestRec object
    find_type $I0, 'Apache::RequestRec'
    r = new $I0

    # who should we say hello to?
    $S0 = r.'args'( )
    $I0 = length $S0
    if $I0 > 0 goto say_hello
    $S0 = 'world'

say_hello:
    # call the puts method to send some output
    $S1 = 'Hello ' . $S0
    r.'puts'( $S1 )

    # tell Apache that we're finished with this phase
    find_global ap_constants, 'Apache::Constants', 'ap_constants'
    $I0 = ap_constants['OK']
    .pcc_begin_return
        .return $I0
    .pcc_end_return
.end

If you are at all familiar with mod_perl or the Apache API, this should look familiar to you, even if you don't know any Parrot. Let's go through the code to see how it works. Because this is not an article about Parrot itself, I'll glaze over the syntax and concentrate on what the code actually does.

The first line of code in the handler declares the namespace in which this handler exists. In this case, it is HelloWorld. This is important because namespaces differentiate one handler from another in httpd.conf.

Next comes the actual handler subroutine, which should always be named _handler. The first thing the subroutine does is to declare some locally scoped "variables." These are actually registers in Parrot, but PIR can abstract them with named variables as it were a higher level language. And ap_constants, a hash that will give access to Apache constants including OK and DECLINED, come next. Both are PMCs, or Parrot Magic Cookie, a special data type that implements the more complex data types of higher-level languages such as Perl or Python.

The code now checks for a query string using the args method of the Apache::RequestRec object and, if one exists, assigns it to the temporary string register $S0. If there is no query string, $S0 will contain world. Next, the code creates the output string, instantiates the Apache::RequestRec object, and calls the puts method to output "Hello World" or "Hello name". By default, the content type is text/html, so there's no need to set it here.

This is the end of the handler, so it's time to tell Apache that we're done and that it no longer needs to handle this phase of the request. This requires returning the Apache constant OK from the ap_constants hash in the Apache::Constants namespace.

To compile the handler into Parrot bytecode, save it in a file with a .imc extension (IMC is short for Intermediate Compiler). Then, compile it into Parrot bytecode (PBC) as follows (this step will be automatic in a future release):

$ parrot -o HelloWorld.pbc HelloWorld.imc

Configuring Apache to Use A Handler

Writing the handler was the hard part. Configuring Apache to use it is easy. The first thing to do is to initialize mod_parrot and load some bytecode libraries:

# mod_parrot initialization
ParrotInit /path/to/lib/ModParrot/init.pbc
ParrotLoad /path/to/lib/Apache/RequestRec.pbc
ParrotLoad /path/to/lib/Apache/Constants.pbc

# our handler
ParrotLoad /path/to/HelloWorld.pbc

ParrotInit tells mod_parrot where to find its initialization bytecode. A future release will probably handle this automatically, but for now explicitly set the path in httpd.conf. ParrotLoad tells mod_parrot to load a bytecode file. In this case, it loads the code that implements the Apache::RequestRec object and the constants hash, as well as the bytecode for the new handler.

Next, Apache needs a location for the handler to, well, handle. How about the location /hello:

<Location /hello>
    SetHandler parrot-code
    ParrotHandler HelloWorld
</Location>

First, this sets the Apache handler for the location to parrot-code. This is the official name of the mod_parrot handler. Then it sets the actual Parrot handler, which, as discussed in the previous section, is the namespace of the handler subroutine, HelloWorld. That's it. Save the configuration, restart Apache, point your browser to http://yourserver/hello (replacing yourserver with the name of your server), and you should see the "Hello World" message. Add a query string to see the output change: http://yourserver/hello?Joe should produce "Hello Joe."

Writing an Authentication Handler

Apache handlers do more than just generate content, of course, and this applies to mod_parrot as well. Here's an example of using an authentication handler to protect a private directory. It will use the HTTP basic authentication scheme, but instead of using a standard password file, it will accept any username as long as the password is "squawk." Here's the handler PIR code:

# this namespace is used to identify the handler
.namespace [ 'TestAuthHandler']

# the actual handler
.sub _handler
    # our Apache::RequestRec object
    .local pmc r
    .local string pw
    .local int status

    # this will contain Apache constants
    .local pmc ap_constants
    find_global ap_constants, 'Apache::Constants', 'ap_constants'

    # instantiate the Apache::RequestRec object
    find_type $I0, 'Apache::RequestRec'
    r = new $I0

    # check the password, ignoring the username
    (status, pw) = r.'get_basic_auth_pw'( )
    if pw != 'squawk' goto auth_failure
    $I0 = ap_constants['OK']
    goto auth_return_status

# authentication failed
auth_failure:
    $I0 = ap_constants['HTTP_UNAUTHORIZED']
    goto auth_return_status

# return our status code
auth_return_status:
    .pcc_begin_return
        .return $I0
    .pcc_end_return
.end

Here is the corresponding configuration in httpd.conf. Instead of using SetHandler and ParrotHandler here, set ParrotAuthenHandler to the namespace of the authentication handler:

<Directory /usr/local/apache/htdocs/private>
	ParrotAuthenHandler TestAuthHandler
	AuthType Basic
	AuthName Private
	Require valid-user
</Directory>

Remembering Why We're Here

Note the low-level nature of the two handlers. There are no else clauses; goto statements appear throughout the subroutine; and return values must be assigned to registers before being used in another operation. You can plainly see that this is only one step above writing assembly here, but remember that you won't have to worry about writing code at this level--you'll write in a high-level language such as Perl 6, and it will eventually compile down to Parrot assembler. Looking forward, the corresponding Perl 6 code for the HelloWorld handler might look a lot like this (as with all things Perl 6, this is subject to change):

use Apache::Constants ':common';
use Apache::RequestRec;

sub handler(Apache::RequestRec $r)
{
    my ($status, $pw) = $r.get_basic_auth_pw();
    return ($pw eq 'squawk') ? OK : HTTP_UNAUTHORIZED;
}

Future Directions

mod_parrot is still in its infancy. It's quite functional, but there is still a lot of work to do before it can power any serious applications. At this point in development, the primary goal is to finish hooking into all phases of the Apache request lifecycle, including support for the relevant Apache API functions. Windows support will also become a priority as mod_parrot becomes more functional.

You may also wonder about CGI scripts. As of this writing, there is no support for running CGI scripts in mod_parrot. mod_perl has Apache::Registry to help CGI scripts run in a persistent environment, and mod_parrot will need a similar infrastructure.

However, the real fun will begin when we have a high level language that we can use to write handlers. If the timelines work out as I hope they will, mod_parrot will be fully functional before the formal release of Perl 6 or any other mainstream Parrot-based language. Because the Apache/Parrot layer will have already been written, this will save quite a bit of development time for mod_perl, PHP, and other similar projects.

If you'd like to read more about mod_parrot, or would like to help with the project, visit the mod_parrot home page.

Perl Code Kata: Testing Imports

Perl Taint Test Kata introduced the idea of Perl Test Kata, small exercises designed to improve your understanding of Perl and your ability to write test-driven code. This article is the second in the series.

Import Testing Kata

Perl 5 added the ideas of namespaces and modules, making code reusable and easier to maintain. To allow convenience, it also added an importing mechanism to put code from a module into the current namespace.

Behind the scenes, when you use a module, Perl loads it from disk and, if successful, calls the special method import(). By convention, this generally imports functions. Much of the time, import() mundanely installs subroutines into the current namespace. That's why so many modules use Exporter to provide a default import().

However, it's also a general module-loading hook that can perform many different types of manipulations. For example, Filter::Simple allows the use of source filters to transform code that looks entirely unlike Perl into valid code in the using module. Other modules change their behavior depending on any arguments passed to import(). This includes Test::More and Test::Simple, which interpret their arguments as information about how many tests to run.

use Test::More 'no_plan';

# or

use Test::More tests => 100;

This feature is both powerful and important. Because of its importance, it needs good tests. Because of its power and flexibility, it may seem difficult to test an import() well. Here are three sample implementations for you to practice testing.

Basic Exporting

package Basic::Exports;

use strict;

use base 'Exporter';
use vars '@EXPORT';

@EXPORT = qw( foo bar );

sub foo { 'foo' }
sub bar { 'bar' }

1;

The tests should check that using Basic::Exports exports foo() and bar() to the appropriate namespace and that they return the appropriate values. Another test is that the code use Basic::Exports (); exports neither function.

Optional Exports

package Optional::Exports;

use strict;

use base 'Exporter';
use vars '@EXPORT_OK';

@EXPORT_OK = qw( foo bar baz );

sub foo { 'foo' }
sub bar { 'bar' }
sub baz { 'baz' }

1;

The tests should check that Optional::Exports exports nothing by default and only those functions named, if there are any.

Load-time Behavior

A few modules have curious behavior. My Pod::ToDemo behaves differently when invoked from the command line versus when used within a module. This makes it substantially more difficult to test. Rather than make you reinvent the tests there, here's a simpler custom import() that does different things based on its invocation. If invoked from the command line, it prints a message to standard output. If used from a module, it exports the same foo() subroutine as before.

package Export::Weird;

use strict;

sub import
{
    my ($package, undef, $line) = caller();

    if ( $line == 0 )
    {
        print "Invoked from command-line\n";
    }
    else
    {
        no strict 'refs';
        *{ $package . '::foo' } = sub { 'foo' };
    }
}

1;

The only really tricky test here must exercise the behavior of the module when invoked from the command line. Assume that the documentation of the module suggests invoking it via:

$ perl -MExport::Weird -e 1

The next page explains some techniques for testing these examples. For best results, spend between 30 and 45 minutes working through the kata on your own before looking at the hints. For more information on how modules, use, and require work, see perldoc perlmod and perldoc perlfunc.

Tips, Tricks, and Suggestions

If you've worked your way through writing tests for the three examples, here are the approaches I would take. They're not the only ways to test these examples, but they do work. First, here is some background information on what's happening.

Reloading

To test import() properly, you must understand its implications. When Perl encounters a use module; statement, it executes a two-step process immediately:

BEGIN
{
    require module;
    module->import();
}

You can subvert both of these processes. To force Perl to reload a module, you can delete its entry from %INC. Note that all of the keys of this special hash represent pathnames in Unix format. For example, even if you use Windows or VMS or Mac OS 9 or earlier, loading Filter::Simple successfully should result in %INC containing a true value for the key of Filter/Simple.pm. (You may also want to use the delete_package() function of the Symbol module to clear out the namespace, though beware of the caveats there.) Now you can require the module again.

Re-importing

Next, you'll have to call import() manually. It's a normal class method call, however, so you can provide all of the arguments as you would to a function or method call.

You can also switch packages, though make sure that you qualify any calls to Test::* module functions appropriately:

package Some::Other::Package;

module->import( @args );

main::ok( 1, 'some test label' );

# or 

::ok( 1, 'some test label' );

Testing Exports

There are at least two techniques for checking the import of functions. One is the use of the defined keyword and the other is through the can() class method. For example, tests for Example #1 might be:

use_ok( 'Basic::Exports' );
ok( defined &foo,              'module should export foo()' )
ok( __PACKAGE__->can( 'bar' ), '... and should export bar()' );

To test that these are the right functions, call them as normal and check their return values.

By the way, the presence of the __PACKAGE__ symbol there allows this test to take place in other namespaces. If you haven't imported the ok() test function into this namespace, remember to qualify it, import it manually, or alias it so that the test program will itself run. (It may fail, which is fine, but errors in your tests are difficult and embarrassing to fix.)

Testing Non-Exports

It's difficult to prove a negative conclusively, but if you reverse the condition of a test, you can have good confidence that the module hasn't provided anything unwanted.

use_ok( 'Optional::Exports' );
ok( ! __PACKAGE__->can( 'foo' ),
    'module should not export foo() by default' );

The only tricky part of the tests here is in trying to import functions again. Call import() explicitly as a class method of the module. Switching packages within the test can make this easier; you don't have to unload the module if you do this.

Testing Weird Exports

The easist way to test an import() function that relies on command-line invocation or produces weird side effects that you may not want to handle in your current program is to launch it as a separate program. There are plenty of options for this, from system to fork and exec to tricks with pipes and shell redirection. IPC::Open3 is one good approach, if you want to use it in your test suite:

#! perl

use strict;
use warnings;

use blib;
use IPC::Open3;

use Test::More tests => 3;

use_ok( 'Export::Weird' );

my $pid = open3(
    undef, my $reader, undef,
    $^X, '-Mblib', '-MExport::Weird', '-e', '1'
);

my @out = <$reader>;
is( @out,                                1,
    'cli invocation should print one line' );
is( $out[0], "Invoked from command-line\n",
    '... with the right message' );

$^X represents the path to the Perl binary currently executing this program. The -Mblib switch loads the blib module to set @INC in the program appropriately. Depending on how you've set up your directories and invoke this program, you may have to change this. The other commands follow the invocation scheme given in Example #3.

Conclusion

You should now have several ideas on how to test import() methods of various kinds. For more details, read the tests of Pod::ToDemo or Test::Builder, which play strange games to achieve good test coverage.

If you've found a differently workable approach, I'd like to hear from you. Also, if you have suggestions for another kata (or would like to write one), please let me know.

chromatic is the author of Modern Perl. In his spare time, he has been working on helping novices understand stocks and investing.

This Fortnight in Perl 6, December 1 - 6 2004

All~

Last week I asked for help identifying the source of a quotation. One friendly soul suggested Alan J. Perlis, but could not find an actual attribution. It did lead me to find a very applicable (and in my mind funny) quote from Perlis, which I will now inflict upon you all, before your regularly scheduled summary:

When someone says "I want a programming language in which I need only say what I wish done," give him a lollipop. -- Alan J. Perlis

Perl 6 Language

qq:i

Jim Cromie wondered if there could be a qq:i which sometimes interpolates and sometimes doesn't depending on whether the variable had been previously defined. There was some discussion which led to the conclusion that this was just asking for strange bugs.

Getters and Setters

John Siracusa wanted to know if Perl 6 would allow one to expose a member variable to the outside world, but then later intercept assignments to it without actually having to switch to using getters and setters in all of the code that uses the variable. The answer: yes, yes you can.

<< foo >>

Richard Proctor asked if he could do <<list of words>>. Juerd pointed out that someone had already asked this. Which brings us to the fine point, ask not Larry for he will tell you both yes and no. Although in this case I think he said "probably"...

Flipflop Operator

Juerd wondered about the fate of the flipflop. Larry explained that while it had lost the election it was still going to work hard for you in the Senate. Err, that's not quite right, he said that "It's leaving syntactically but not semantically.", but he hasn't specified the new syntax...

Temp $var

Alexey Trofimenko wanted to know whether temp would preserve or destroy its old value. Larry is leaning toward the Perl 5 semantics of destroying, I think.

State vs. My

Alexey Trofimenko wondered how much advice about optimizing Ruby also applied to Perl. Unfortunately, he also misunderstood the state specifier. The topic then quickly veered into what exactly state does.

Specifying a Hash's Key Type

Abhijit Mahabal wanted to know if he could specify a hash's key type. The answer is yes, but the exact syntax seems to be worth a discussion. Luke Palmer, in his Mathematician's rage, attempted to shoot down any usage of Domain and Range, as they really should be Domain and Codomain.

Wikipedia: range Range (Mathematics)

Container Methods

Ashley Winters wants to have syntax for calling a method on the container object rather than the containee. Luke Palmer agreed that this was problematic. Larry appears to be in no hurry to add more operators for this one, yet.

Slight Discrepancy Between Synopses

Stéphane Payrard pointed out a small issue in some Synopses. Larry replied "oops".

Arrays, Lists, Iterators, Functions, Co-routines, Syntax

Many people suggested many things about the best thing to replace the now missing <> op. I think Larry is leaning toward adding an undare = op, which would do cool things. I don't thing anything is final yet.

Iterators as Functions

Unary = Talk

Push/Pop/Pull/Monkey

Many folk voiced their dislike of shift and unshift. I must agree with them, but they also suggested a great many alternatives, including pull/put, get/unget, and even getting rid of push/pop. I must say that I really dislike that last idea. Fortunately I am not alone. Currently we are waiting for inspiration to strike.

Topicalization

Someone noticed that for might override one's topic at undesired times. Larry ruminated about ways to solve this.

Required Whitespace

Rod Adams does not like inconsistent whitespace rules. Larry explained why the existing rules really were consistent.

Perl 6 Compilers

The lack of traffic on p6c has given me another space to abuse. You should listen to "Soul Coughing." If you would like to join in the fun of abusing p6c, you should submit tests. Nothing is more abusive then stress testing. ;-)

Parrot

Tuning and Monitoring

Matt S asked how much support Parrot has for tuning and monitoring. This week I exercise the awesome powers of the summarizer and invoke the mighty Warnock's Dilemma.

IMCC Globals

Leo removed some IMCC globals. Nice work.

Ensure Directories Exist First

Andy Dougherty fixed a problem with writing a file in a non-existent directory. Leo applied the patch.

Namespace-sub Invocation

Last week Luke Palmer wanted to know about calling subs in namespaces. I posted Leo's answer, but Dan does not like it. It should be a two-step process: first fetch, then invoke.

What Is an opcode?

Thomas Seiler attempted to clear up some perceived confusion about what exactly an "opcode" is. No responses...

Lexicals, Continuations, and Register Allocations

Dan voiced a final word in this long-lived and lively thread, which kicked off several children. Return Continuations (even once-promoted) restore their registers.

Dan's Ruling

The Long and Lively Thread

Keyword Arguments

Sam Ruby wondered how he ought to handle keyword arguments to functions. Dan admitted that this is complex, and outlined the cheat he has been contemplating. No one has either commented on or implemented it yet.

What Is and Isn't Up for Grabs

Dan attempted to lay out clear rules as to what things he would entertain until Parrot was functionally complete. Let's hope it sticks.

AST + COMPILE_IMMEDIATE == :-(

Bernhard Schmalhofer provided a patch to fix some of the :-(. Leo applied it.

t/dynclass/pybuiltin.t Fails

Will added a BUG to RT for this.

Benchmark Tests

Justin DeVuyst submitted a patch, which fell through the cracks, to make the benchmarks also be tests. Fortunately Matt Diephouse rescued it from the cracks and Leo applied it.

Too Many opcodes

Leo voiced his opinion that there were too many opcodes and suggested a scheme for cutting down on them. Dan corrected him that there were not too many. Despite this surface disagreement, however, Dan addressed the spirit of Leo's complaint. Thus, they can both be happy.

C89 Issues

There was a little confusion about whether variable declarations could follow code in C89. The answer is not.

perlhash iter busted

Sam Ruby noticed that the perlhash iter did not work and submitted a test case for it. Leo fixed it (and presumably applied the test).

Warnings Cleanup

Garrett Rooney submitted a patch to fix some warnings. Leo applied it.

Objects, Classes, Metaclasses, and MMD Dispatch

Dan's attempt to spec out objects from last week led to some discussion of MMD. Leo suggested an implementation. Silence reigned.

More MMD Stuff

More Tcl Stuff

Will added more new stuff to Tcl. Yay, Will!

Internal Exception Clean Up

James deBoer submitted a patch that cleans up internal exception output. Later he submitted a second, better version of the patch. Warnock applies.

Dynamic Evaluation of PAST

Bernhard Schmalhofer submitted a patch to add support for PAST dynamic evaluation. Leo applied it.

Inline Caching

Leo explained inline caches and why 19 out of 20 calls like them. This led to his suggesting that some of the opcodes we have that take offsets from strings are a premature optimization. This led to some discussion about whether what he suggested was in Dan's earlier mandate.

Inline Caching

Leo's Suggestion

help Target in Make

Bernhard Schmalhofer (whose name makes me very thankful for copy and paste) submitted a patch adding a new help target to parrot/docs/Makefile. Will appied it.

Parrot and Strong Types

Cameron Zemek asked about supporting strongly typed languages on Parrot. I confused strong-typing and static-typing, but fortunately Dan came to the rescue with a good explanation. During the course of this thread, people suggested both Haskell and Prolog on Parrot. I like them both, a lot.

Original Message

Dan to the Rescue

Move libnci.def Out of Root Directory

Mitchell ::mumble:: provided a patch to do the above some time ago. Will rescued it and asked for a ruling. The ruling is that it should apply. I don't know if anyone DID apply it, but someone should.

Tru64 cc/ld Issues

Jarkko Hietaniemi posted a problem with building on Tru64. Sam Ruby committed it.

The Usual Footer

If you find these summaries useful or enjoyable, please consider contributing to the Perl Foundation to help support the development of Perl. You might also like to send feedback to ubermatt@gmail.com.

The Evolution of ePayment Services at UB

In the summer of 2001, the State University of New York at Buffalo (UB) hired a new Provost. She surveyed various school services and came up with a short list of must-do projects. Given the level of competition in higher education, it's no surprise that she felt that we must be able to accept payments over the internet. In general, we try to make it as easy as possible for students and their parents to manage the business side of their education. Electronic payment is a key, new component of that.

How to Start?

Determining how to accept an electronic payment requires skills from a wide variety of groups on campus. So we first put together a committee with representatives from Accounting, Student Accounts, and some offices that wanted to accept electronic payments. My department, Administrative Computing Services, also sat on the committee since we would have to implement the actual technology. Our role was to work with the people who handled the manual payment system and figure out how to automate that system through the web.

We had no experience taking money over the internet, so we looked at our options:

  • Purchase a fully outsourced solution.

    We evaluated many all-in-one systems, but most didn't do exactly what we wanted. The biggest problem was integration with our existing system, which had grown over time and had many legacy components. Many vendors had systems that worked only with their complete accounting systems.

  • Partially outsourced solution.

    With this approach, we would do some work on our end and offload some to a vendor. This ended up being our initial approach since it allowed us to write the custom parts to interface with our systems, but still outsource the financial parts to reduce our initial risk.

  • Mostly custom solution with payment processing outsourced.

    With this configuration, we'd host all of the web components in-house. We would use an external vendor for the payment-processing components, including validating credit cards, encumbering the funds from cards, and processing the actual payments through the payment networks and banks.

  • Complete in-house system.

    Some companies actually act as their own payment processor in addition to running the front-end website components. There is nothing preventing you from processing your own payments, but we found the banking world incredibly complex, especially with regard to credit card transactions. Although we could theoretically save money by doing this work ourselves, we weren't interested in doing this work. Our Accounting department agreed.

After evaluating the options, we chose to start with a partially outsourced solution. Over time our needs changed and we became more knowledgeable; we since made the transition to a mostly in-house solution.

Initial Solution: Partial Outsourcing

Our initial solution was mostly outsourced, with us building just a few front-end components. This allowed us to get our feet wet with electronic payment without running everything. Even with a vendor doing most of the work, we encountered some challenges.

To implement this model, we built a website with a few pages that would collect personal data and track some data on the transaction. On the final page of our site, the user clicked a submit button and we shipped them off to the vendor website to enter the credit card information. This way we didn't directly deal with any credit card issues. Each night we would receive a remittance file with the day's transactions and we would reconcile these with our data using a unique identifier.

The key issues for this first phase were:

  1. Purchase and install a certificate from a widely accepted certificate issuer.
  2. Make sure all users used SSL. We simply redirected all http requests to the site to https so users couldn't accidentally use unencrypted connections.
  3. Accurately record a unique identifier to match up transactions when we received the remittance file from the vendor.
  4. Follow the vendor's protocol to integrate properly with their site.
  5. Understand the payment-processing system so we could date and post transactions accurately in our batch jobs.

The first two were basically normal tasks for setting up a secure server. The third required a little work, but not much. The challenge was integrating with the vendor. They had limits on the size of a unique identifier that were smaller than we wanted to provide, so we needed to shorten the value we sent. We also had concerns with how we would track down any "orphan" payments in the payment system that we couldn't track to a particular user. We were lucky that, in practice, this never occurred.

Number four also presented a challenge. My department is largely a Perl shop and we didn't use Java on our website. During the development process, we discovered that the vendor link required Java to encrypt the data we sent to their system. They used Cryptix PGP with some custom code, available in Java only. That summer at OSCON I learned about the Inline modules and I decided to try one. Since it was only one small Java component, I followed their examples to code the Java encryption subroutine and stuck it at the bottom of the final Perl submission script using Inline::Java.

We have since moved on to a different system for processing our payments, so this code is a bit dated. However, when we were using it, it looked something like this:

my $encrypt = new encrypt();

# We had several calls like this one, one for each parameter we
# needed to encrypt.
# Each call returned a string with the encrypted data.
my $vendor_code = $encrypt->send_to_vendor($pay_code);

# CGI code to create a form with the encrypted data as hidden fields.
# The form also acted as a confirmation page, so the user saw
# "Please review your information and click to continue".

# Java code at the bottom of the file.
use Inline ( Java => <<'END_OF_JAVA_CODE',
  import xjava.security.Cipher;
  import cryptix.pgp.*;

class encrypt {
  public encrypt(){
  }

  public String send_to_vendor (String value_string){
      PGP pgpObj = new PGP();

      /* Initialize the object with base values. */
     pgpObj.init("/pgp_dir", "client name", "passwd","sender_key_name");

      /* Add the receiver name to the list. */
      pgpObj.addReceiver("vendor name");

     /* Create the encoded string. */
     String encvalue = pgpObj.encodeString(value_string);

     return (encvalue);
   }
}
END_OF_JAVA_CODE
SHARED_JVM => 1,
CLASSPATH => 'paths to cryptix libraries',
);

The code itself is very simple. The Java code simply creates a new object, calls a few methods, and returns the encrypted string. The interesting part is how relatively easy it was to embed the Java code in a Perl CGI script and have it work seamlessly. When we initially implemented this in 2001, Inline::Java was fairly new and there was little documentation on how to do this. Patrick LeBoutillier, the author, was very helpful as I was trying to get it working. Now the Inline::Java documentation has a section on using Inline::Java in CGI scripts.

The last item on the list was far more difficult than we anticipated. For most days, it was a simple formula to tag a record with a date the money would be available. However, weekends had special rules; we saw results that were inconsistent with the documentation we were working from. To nail down the problem, we ended up having to run small test transactions at various times on various days to see when our accountants saw the money actually arrive in the UB account.

For example, if the documentation said there was a 4 p.m. cut-off time for payments on a given day, we would run a $1.11 transaction at 3:55 p.m., then another $1.12 transaction at 4:05 p.m. Our accounting people would watch the system to see when the payments came through. We needed to increment the amounts because we used the same credit card to process the transactions and there was no other way to keep them straight.

Following this method, we learned a lot about the backend banking system. Different credit card companies process things on different schedules and handle things differently. The most important thing we found was a bug in the vendor's processing system. After many weeks of running test transactions and comparing the clearing times with the vendor documentation, we finally tracked it back to a problem in the vendor system that they quickly fixed.

The final product worked well. During the first run of the final confirmation script, Inline::Java compiles and saves the code to run again, so it was quite fast. The vendor received the encrypted information correctly and could decrypt it successfully; it didn't matter to them that we actually generated the request in Perl. Our remittance files arrived on a regular schedule and we posted the results on a daily cycle.

The World Happens in Real Time

As described above, the vendor did much of the work, but since we passed users off to their system, we were unable to see the results of transactions until the next day. This wasn't a problem for some payments. For example, the graduate school application fee just needs to be received by a particular date. It's not crucial that we know the results of the transaction in real time.

For tuition payment, however, we needed a real-time solution. At UB and many universities, you can't register for your next semester if you have an outstanding balance from the previous semester. When a student is in this situation, called a checkstop, he needs to be able to pay and have the checkstop lifted immediately. Students often find out about their checkstop while trying to register online, so they need to be able to pay, clear their accounts, and return to the registration system while some courses are still available. What's more, this often happens on a Saturday morning when many services may not be available.

Real-Time Integration Challenges

The existing accounts system at UB runs on a mainframe, but most new development happens in our distributed Solaris environment. To allow real-time payments, we needed a vendor who could provide an immediate payment response, and we needed to apply that result to the mainframe. We also wanted to design the system to be open for future applications.

We selected VeriSign as our new vendor and they returned real-time results. To then apply real-time results to our system, we could no longer pass users off to another site; we needed to manage the entire transaction. We redesigned our system to handle this.

We currently use this redesigned system. The key steps in our current process are:

  1. Manage the web transaction using Perl CGI scripts and an Oracle database.
  2. After the user enters credit card information and clicks Submit, make a SOAP call to our internal payment server. The payment server logs all events.
  3. Perl scripts on the payment server call our mainframe system to determine if the student's account is eligible to receive a payment.
  4. If the student has an active account, we call VeriSign using their Perl module.
  5. Receive and log results from VeriSign.
  6. If VeriSign approved the payment, call the mainframe again and credit the student's account with the amount of the payment. If student paid in full, remove the checkstop from the account.
  7. Return the response for the initial SOAP call and present the user with the result of the transaction.

From a design perspective, most of the steps are necessary given the disparate components we're working with. Despite all of the things we do, the entire process usually takes about 10 seconds.

We have several systems that access our mainframe in real-time through Perl. The Perl part is a basic socket call to a port on the mainframe. Special software runs on the mainframe as a server that can receive formatted requests and return results after processing against mainframe systems. One major advantage of the mainframe software is that it allows us to use existing mainframe systems through a new internet front-end.

The interface with VeriSign was likewise fairly easy to integrate. They provide a Perl module to call their system and retrieve results. The majority of the work was in learning all of the non-success response codes and designing the system to handle them appropriately.

However, you may wonder why we included a SOAP call between internal servers. When we designed the system, we tried to make all of the parts as flexible as possible. For this phase, we built both the front-end (the user websites) and the backend (the payment server). Given a generic enough interface, we speculated that other campus users may want to build their own front-end website and interface directly with the payment server. Constructing a SOAP interface using SOAP::Lite allowed us to potentially provide a service in a technology-neutral manner. The system has worked well and performance hasn't been a problem. Thus far, we haven't had an external, campus-entity interface with our system, but we have built other internal systems that use the interface.

We also have a process in place for payments that don't need immediate application (not real time). For these payments, we follow a similar process but skip the live mainframe connections. At night, we have batch Perl jobs that run and apply the changes to the mainframe.

Challenges

Most of our challenges with the ePayment system are with the user interface, specifically providing enough information. We will frequently receive a result of "declined," or "declined with possible approval." Some declines are the simple type where the user simply doesn't have enough room on their credit card. However, many credit and debit cards have a maximum charge limit per day that card holders aren't aware of. They tend to hit this limit when using a credit card to make a large charge like a tuition payment. This type of decline is conditional; the card issuer can approve the transaction after a phone call from the card holder.

The problem is that the return code from VeriSign isn't always clear enough for us to tell users exactly what they need to do. The correct response can also vary from one credit card issuer to another. The challenge is presenting them with messages that give them enough information, but no incorrect information. Even when we can provide them with a clear message such as "You need to call your credit card issuer," they often assume it's a problem with our system and contact us anyway.

Your eUniversity

UB's ePayment project has been very successful and our users have received it well. In our last payment cycle, approximately 40% of payments came through the internet, instead of the traditional methods of mailing a check, going to the payment office, or paying over the phone. Even with this level of activity, the level of support required has been very reasonable.

With so many internet payment systems available -- some specifically designed for education -- some may question custom-building so much of the system. When dealing with legacy components, however, it's often difficult to purchase a solution for just part of the process. Replacing entire systems at a university the size of UB is a much larger project than we usually want to tackle at any one time. In these situations, we've found Perl to be very flexible, allowing us to patch together disparate systems and external resources to build a cohesive, effective system.

This Fortnight in Perl 6, November 16-30 2004

All~

Welcome to yet another summary. Although Aliya is present for this summary, I think the unnamed gecko with its tongue out will be the one who is helping to bring it to you, aided of course by Nicola Conte and Massive Attack. Rather than try to do something witty about the strange music I am listening to, or the stuffed animals who are assisting me, I will start this summary off with an entirely self-serving request. <abuse>A while ago I saw the quote, "Computer Science is merely the post-Turing Decline of Formal Systems Theory," without an attribution. I have tried to find an attribution for it, but have been unable to find one. If any of you know it, that information would be appreciated.</abuse>

Without too much further ado, I give you Perl 6 Language.

Perl 6 Language

Return Values from a Substitution

Nicholas Clark wondered what kind of return value the s/// operator would return. Larry provided the answer: a match object that does wicked, smart things based on context (Boston is getting to me).

Deep Operators

Matthew Walton wondered about deep operators and return types. No answer yet, but it is still a little early to call in the awesome forces of Warnock.

Perl 6 Compiler

The race between Google and the compiler is over and Google loses. Badly. While, alas, I do not have a pretty interface, you all have links from nntp.perl.org, plus a nifty infant grammar engine to torture.

Parrot Grammar Engine

Patrick R. Michaud (forever after known as Patrick) released a first version of the Perl 6 Grammar Engine (P6GE). Sometime later, he renamed it the Parrot Grammar Engine, as it will be standard in Parrot and PGE sounds better. He asked for people to poke and prod it. Many people did, with quite some glee. Some work has been done adding it to the Parrot build and making it use Parrot's Configure/make system, as well as some cross-platform cleanups. Patrick also put out a request for tests and explained the basic framework already in place. He also explained his short term plans.

Synopses and Apocalypses

Larry has managed to persuade dev.perl.org to host the latest and most up-to-date versions of the Synopses. Consider the former link deprecated. Larry also mentioned that this should probably wait a little while before it hits Slashdot. Dan put a desperate plea not to also, as "I feel obligated to actually *read* the comments on Parrot and Perl 6 stories on Slashdot, at 0."

Parrot Internals

Having fought my way through the lack of Google, I can now return to its calm, warm tones and convenient threading.

RT Data Extraction

Will Coleda pointed out that he has the mystical power to extract and summarize data from RT automatically. Then he asked for suggestions about how to use this power for good or for awesome.

Locating Shared Libraries

Adam Warner has some trouble locating shared libraries on Debian. Leo pointed out that Debian apparently names these libraries funny things. The he added support for symlinks and asked Adam to provide the appropriate symlinks.

Double v. Float

Adam Warner played around with the Leibniz summation for pi example, but could not get it to be more precise than 3.1431591. It turns out that the print opcode was at fault and that sprintf worked better. In fact, it allows you to print more precision than you actually have. Ain't life in the floating point world a pain? We should all use integers, where pi is exactly 3.

AIX PPC JIT

The ongoing thread of three-letter acronyms finally seems to have run its course, ending in a climactic upgrade to Perl 5.8.5.

New Calling Scheme Proposal

Leo posted a new, calling scheme proposal. Dan stomped on it hard, then posted an updated version of PPD03 (calling conventions). People spent some time fleshing out every little corner of the new doc.

Parrot BASIC and Perl.org Blacklists

Joshua Gatcomb kindly acted as a intermediary between the list and Clinton A. Pierce. Apparently perl.org has blacklisted Clinton's domain and ISP. He would be willing to resume work on it, but he would like help, and/or to be un-blacklisted. There was no real answer to Clinton's troubles. On a side note, my old email provider, www.softhome.net, also occasionally gets blacklisted by perl.org (it seemed to go in fits and starts).

CVS Access for Parakeet

Michel Pelletier was wondering if he could have CVS access to the Parakeet language directory now that he has a perl.org username (michel). Dan said that he had forgotten to mention that he had put in a request, and he would likely forget to mention when the request went through.

Return with Return Continuation

In the confusion of what not to change and what to change, the op to return to the caller by invoking the current return continuation disappeared in the shuffle. After sorting out the fact that Dan wanted it to go in, Leo added it, naming it returncc for lack of anything better.

Parakeet Broken

Jeff Horwitz noted that Parakeet did not quite work. Leo pointed out that Parrot had outflown Parakeet and it needed to catch up with the new eval changes.

Deprecation of P0, P1, P2

Leo sent a warning to the list that, as per PDD03, access to the current continuation and sub should only come from the interpinfo ops. Usage of P0 and P1 for these things is officially deprecated and soon to be discontinued. Also, P2 current object came along for the ride later, which caused troubles with imcc's self macro. Later, he removed the deprecated things.

Main Is Just a Sub

Leo has coerced main into looking and acting exactly like any other sub. Consistency is nice like that.

Continuations, Basic Blocks, and Register Allocation

The thread of the week last week continued strong into early this week, slowly but steadily clearing up misconceptions of certain summarizers who would like to remain anonymous. Also, later in the week, Bill Coffman provided a summary of the problems and ideas thus far. We all eagerly await the resolution.

m4 0.0.10

To keep up with the aforementioned eval changes, Bernhard Schmalhofer provided some m4 improvements.

Lightweight Calling Conventions

Dan, Leo, and Patrick had an interesting discussion about the speed of various calling conventions. While BSR/RET beats continuations, continuations with tail-call optimization beat BSR/RET; however, BSR/RET with tail-call optimization edges out continuations. In the end, the PGE will be agnostic to calling conventions so as to provide a real-world test for benchmarking. Either way, fear the Big Sweaty Russian (ask Mike Z if you want to know ;-).

NCI Signature Correction

Last week I incorrectly claimed that the d signature allowed access to a raw buffer. I was wrong, it was b. Thanks for the catch, Bernhard.

Void Functions Don't Return Things

Andy Dougherty noticed that matchrange.pmc attempted to return something from a void function. He supplied a patch, but there seems to be no response....

Perl6 --tree

Gerd Pokorra submitted a patch to fix a problem with perl6 --tree. Warnock applies.

ResizableIntegerArray Needs Soda

Patrick requested that someone provide the wonderfully amazing ResizableIntegerArray pop. Once that is done, PGE should switch from using a PerlArray to a ResizableIntegerArray. Patches welcome.

No perldoc => WARNING

James deBoer provided the previously requested patch for doing the right thing when there is no perldoc present. Leo applied it.

Broken Benchmarks

The great thing about breaking benchmarks is that if you do it right, your execution time goes from a big number to nearly nothing. It provides one hell of a speed up. Unfortunately, it also means they don't work. Alas, such is life. Fortunately Leo provided an explanation, if not a fix.

MMD with Native Types

Leo voiced his confusion about MMD_ADD_INT. When Leo is confused, I worry.

Polymorphic Inline Caches (PICs)

Leo posted some interesting stuff about PICs and provided some preliminary benchmarks. He also provided a link to the article from which he drew the idea.

Mac OS X.2 Failures

Klaas-Jan Stol noticed that pmc2c2.pl failed on his 10.2 installation. He asked for help, but no one has responded yet.

Inconsistent opcode Names

William Coleda noticed that we did not consistently use_underscores or pushwordstogether. Everyone agrees this is a problem, so we are probably just waiting on a brave soul who can make the necessary sweeping change. I think that underscores_won_out.

Tcl Supports Lists

Will Coleda added basic list support to Tcl. Then he threatened to do more with it. I, for one, welcome our new list overlords.

Build Problems on PPC Linux

chromatic (whose name Google rudely capitalizes) had some trouble building Parrot on PPC Linux. No resolution yet.

New TODO Items

Will Coleda added some new TODO items.

Intellectual Property

Tim Brunce asked how Parrot was handling IP concerns. Dan told him that we have lawyers working up "Real Paperwork" even as we speak. In the meantime, it is mostly an honor system.

1 - 2 == BOOM

Will Coleda found a bug in MMD. Leo took responsibility and submitted a fix.

Escaping Strings

Patrick wondered if some code sitting around would properly escape a string into a nice, clean form. The answer is yes, use Data::Escape.

Bug in Method Calling with nonconst Keys

Luke Palmer found this bug. Leo fixed it.

Underscores on Subs

Luke Palmer wondered if subs still had to have _ before their names. Leo provided the answer: no, and the docs are out of date.

Silent Effect of opcodes

Leo noted that opcodes with silent effects need a little more special treatment. This morphed into a conversation about the continuation troubles that have been haunting us all and exceptions, too.

IMCC Tracing, Leaving Subs

Will Coleda asked if it would be possible to indicate what route Parrot would return to after a returncc opcode. Leo liked the idea and put in a quick hack for it, which needs cleanup.

Threads, Events, and Win32 (Oh My!)

Gabe Schaffer continued to explore the problems and approaches to portable threading with Leo's help. Best of luck, guys.

IMCC Register Mapping

Will Coleda noticed a possible optimization with register mapping. Leo said that it was not that simple, but it would be implemented at some point in the new register allocator.

Exceptions

Exceptions hurt my head, especially Dan's description of them. Thus I will just leave you to read it for yourself, so your head can hurt, too.

Missing Makefile Dependencies

Leo noticed that the Makefile is not quite right about its dependencies. This frequently recurs.

JITted vtables for Sparc

Stéphane Peiry provided some more JIT support for the Sparc. Leo applied the patch.

PGE Namespaces and Names

Will Coleda, Luke Palmer, Jerome Quelin, and Patrick all worked to clean up PGE to have its own namespace and consistent naming.

Parrot_compreg Signature

Bernhard Schmalhofer noticed that Parrot_compreg had a different signature than documented, so he submitted a patch to fix it. Leo applied it.

opcode Numbering

Leo added a TODO ticker for opcode numbering.

Tests Pass, but Create Core Files

I reported that on my machine all of the tests claim to pass, but core files appear in the parrot directory. Dan confirmed my suspicion that this was a real problem. I tried to supply helpful information like back trace, but Warnock applies.

pcc Cleanup

Leo removed the recently deprecated P0, P1, P2 usage. Relatively few tests break as a result.

Namespace-Sub Invocation

Luke Palmer wanted to know if there is syntax for calling subs from a particular namespace. Leo provided the answer: call it as a method on the namespace PMC.

Reserved Words Annoy

Luke Palmer wondered if he could define a .sub "new" to get around reserved word problems. Leo added the support, but warned him not to use "spaces or such."

Lexicals, Continuations, Register Allocation, and ASCII Art

This thread (and the ones that preceded it) have made me wish that Gmail and Google Groups had a fixed-width font option. Sadly, this summary will probably not get me it, as it did not get me p6c either. Ah well. The problems associated with lexicals and continuations churned. There was a plea for guidance from Dan. Hopefully he will guide us shortly.

Old Parrot Question

Bloves had a question about assemble.pl in the Parrot 0.0.1 source. Warnock applies.

Parrot on Linux PPC Build

chromatic managed to track down some of the PPC build issues for Linux. Leo wondered where all of the config hackers went. I think Leo wants someone to fix it.

Preprocessor Problems

Flavio S. Glock noted that the preprocessor did undesirable things. Leo suddenly realized that it was woefully out-of-date and need major fixing/disabling. Flavio submitted a small patch for it, but Leo pointed out that this was not enough to actually fix the preprocessor.

New push_eh opcode

Leo added a new push_eh opcode and deprecated the old set_eh.

TODO and RT Wrangling

Will Coleda continued his stellar work managing the RT queues. Brent "Dax" Royal-Gordon patched the first one.

PMC Does "hash|array|monkey"

Will Coleda added a ticket for the does functionality. He promptly closed it when Leo pointed out that it already existed in the form of the does op. So Will up and started using it.

Strings Are In

Dan announced that the last of his string stuff was checked in and he would like brave souls to help him merge it back into the tree.

Language TODO Tests

Will Coleda thanked Josh Wilmes for fixing the Language TODO tests. Apparently one of Tcl's has mysteriously started working.

Minesweeper Broke

Jens Rieks noticed that somewhere in the shuffle minesweeper broke. As a professional minesweeper-er, this shocks and offends me. Leo offered a pointer as to where to start looking to fix it.

@ANON Subs

Leo added some support for anonymous subroutines. People liked them and began to use them. Luke Palmer promptly tried combining it with @IMMEDIATE and began to wonder if it were immediate enough.

Parrot Grammar Engine Issues

I think that I shall continue to spell out PGE for a little while, but I may decide to use only the acronym in some later summary without warning, so be prepared. Oh right, the whole reason I mentioned it in the first place ... Will Coleda pointed out a few issues, including (1) doesn't compile by default and (2) doesn't compile at all. Nicholas Clark fixed (2), and Patrick figured that (1) could wait until it was a little more mature. Personally, I feel that maturity is overrated, poo-poo head.

(Java|ECMA)Script on Parrot

David "liorean" Andersson wondered if there were any projects to run JavaScript on Parrot. While there does not appear to be, many people think it would be a good idea and offered useful pointers.

Rules Engines

Cindi Jenkins posted a link to an interesting blog entry on rules engines. Unfortunately, I think she posted it to Google Groups as it did not find its way onto the list. Also, unfortunately, it is a post about logic rules engines a la Prolog and not grammar rules. Who knows, though, there might be some overlap....

PIR Examples on the Website

Herbert Snorrason noted that there were no PIR examples on the website and opened a ticket for it. Will took a first pass at it.

testj string_102 Issues

Luke Palmer found a subtle-looking problem in string_102.pasm. Leo couldn't reproduce it and suggested that it might be a Red Hat thing. Peter Sinnott chimed in with a possibly unrelated failure, also on Red Hat.

Tuning and Monitoring

Matt S. posted a question about how much internal tuning and monitoring Parrot will allow. Unfortunately, I think he posted it to Google Groups, as it didn't show up in my inbox. I am honestly not sure how much is available or in the works.

Missing directory in MANIFEST

Andy Dougherty submitted a patch fixing a missing directory in the manifest by allowing ops2pm.pl to add it.

Objects, Classes, Metaclasses, and Tall People's Heads

Dan posted a refresher on the current state of the object system. He then went on to explain where it was headed and conjectured that this would be enough. Just reading his description of it hurts my head enough that I do not wish to guess if he is right or not.

eof opcodes

Brian Wheeler wondered why there was no "eof" opcode. Leo told him that it was a method on a PIO object. Apparently most of the IO opcodes should actually be PIO methods. Go figure.

Ops to Deprecate or Not

Leo attempted to trim down the core by removing some opcode variants. Dan did not appreciate the change and told Leo to roll it back. Fortunately, Dan went on to explain the longer-term goal (which should satisfy Leo).

Perl 6 Language

Deep Operators

Last week, Matthew Walton wondered about Deep Operators and if they would work as he expected. As I tentatively predicted, the answer came back and was "yes." Then the thread got sidetracked with context and Perl vs. perl vs. PERL concerns.

Gather to Separate Bins

Dave Whipp wanted to know if he could use gather/take with multiple bins. Michele Dondi suggested using adverbs for it. Rod Adams pointed out that as gather is inherently lazy, the two-binned approach could cause strange results (like churning for a while, filling one bin, and trying to get an element for the other). Of course, not being able to do it would mean possibly having to compute an expensive generator function. This is more than necessary, unless it is memorized....

<< In Here-Docs

Thomas Seiler has decided to test my copy-paste-fu by starting a discussion on characters that don't appear on my keyboard. His question was if <<END could start a here-doc. The answer appears to be "no."

« != <<

The above thread led to a discussion of the various quoting operators, and the differences between «» and <<>>. This led to much discussion on the finer points of qw, qx, and qq (among others). Juerd suggested scrapping qx and qw in favor of qq:x and qq:w, which Larry liked. Rod Adams suggested scrapping <<END in favor of qq:h/END/, which I like.

Unifying Namespaces

Alexey Trofimenko wondered about unifying the $, @, and % namespaces. Larry told him that this ship sailed long ago and that it was not changing.

Lexing/Parsing Perl6 Without Executing Code

Adam Kennedy wants to be able to syntax color (and possible perform basic programmatic manipulations of Perl 6 code). Anyone who has used a good IDE like Visual Studio, Eclipse, or IntelliJ probably understands where he is coming from. He does not, however, want to execute arbitrary code in the process of doing this. Much discussion ensued, but it does not look like he will be able to do this. Quite the tragedy, but Perl is simply too dynamic for the docile lexing of static languages.

Perl6 Compiler

Nice Work

Nick Glencross has the honor of the only message on p6c this week, but his sentiment is shared by all. Nice work, Patrick.

The Usual Footer

If you find these summaries useful or enjoyable, please consider contributing to the Perl Foundation to help support the development of Perl. You might also like to send feedback to ubermatt@gmail.com.

Building a 3D Engine in Perl


This article is the first in a series aimed at building a full 3D engine. It could be the underlying technology for a video game, the visualization system for a scientific application, the walkthrough program for an architectural design suite, or whatever.

Editor's note: see also the rest of the series, events and keyboard handling, lighting and movement, and profiling your application.

First, I'll set some goals and ground rules to help guide the design. I'm all for agile programming, but even the most agile development process needs some basic goals at the outset:

  • I'm not going to make little demos. Early on, the engine won't have much functionality, but it should always be a good foundation for future growth.
  • The engine must be portable across architectures and operating systems. I will use OpenGL for 3D rendering and SDL for general, operating system interaction, such as input handling and window creation. The engine itself should contain almost no OS-specific code.
  • The engine should be operational at every step of the way, from the very beginning. I will flesh it out over time, and there may be some complex concepts that take some time to work through, but at the very least, every article should end with the whole engine working again.
  • I'll leave out most error checking to save space and make the central concepts more clear. For the same reasons, there is no included test library. In your own engine, you will want to have both!
  • Don't be afraid to experiment. The best way to learn this stuff is to play with it. Start with what's in the articles and add to it. Any time you spend now will repay itself many times over later, because it's easier to understand advanced topics when you have a solid understanding of the earlier topics.

As a final note before we begin, some versions of SDL_Perl have bugs that will affect the engine. If I know of any issues to watch out for, I'll let you know; conversely, if you find any bugs, let me know, and I'll include a note in a following article.

Getting Started

The first step is to rough out a simple structure and make a runnable program right off. Bear with me; there's a fair bit of code here for what it does, but that will simplify things later on. Here's my starting point:

#!/usr/bin/perl

use strict;
use warnings;

my ($done, $frame);

START: main();

sub main
{
    init();
    main_loop();
    cleanup();
}

sub init
{
    $| = 1;
}

sub main_loop
{
    while (not $done) {
        $frame++;
        do_frame();
    }
}

sub do_frame
{
    print '.';
    sleep 1;
    $done = 1 if $frame == 5;
}

sub cleanup
{
    print "\nDone.\n";
}

The first few lines are the usual strict boilerplate, especially important since I'm working without a net (a test library). Then I declare a couple of state variables (a "done" flag and a frame counter), and jump to the main program.

The main program is pretty simple -- initialize, run the main loop for a while, and then clean up. It's typical of how I structure a potentially complex program. The top-level routines should be very simple, clear, and self-documenting. Each conceptual piece is a separate routine, wherein reside all the gritty bits that actually do the real work. I've seen huge programs (hundreds of thousands of lines) where the main procedures started with several hundred lines of initialization before finally branching to the "real" main body at the end. That style is hard to debug, hard to profile, and just plain hard to understand. I avoid it religiously.

Back to the program at hand. init sets autoflush on STDOUT so that partial lines print immediately, which I use later in do_frame.

The main_loop simply loops until $done is true, producing one finished animation frame per loop. Each loop increments the frame counter and calls the actual routine that does the work, do_frame.

do_frame prints a single dot to indicate a frame has begun, and sleeps for a second. When it wakes up, it checks if five frames have completed, flagging $done if so.

With $done set, main_loop ends and control returns to main, which calls the final cleanup. cleanup just notifies the user of a clean exit and ends.

That's a fair amount of code to print two lines of text (over the course of five seconds) and exit; it doesn't even open a rendering window! I'll do that next.

Creating a Window

First, I need to pull in the SDL and OpenGL libraries:

use SDL::App;
use SDL::OpenGL; 

and add a couple more state variables (a config hash and an SDL::App object):

my ($conf, $sdl_app); 

Initialization

I'm going to do two new types of initialization, so I create routines for them and call them from init:

sub init
{
    $| = 1;
    init_conf();
    init_window();
}

sub init_conf
{
    $conf = {
        title  => 'Camel 3D',
        width  => 400,
        height => 400,
    };
}

sub init_window
{
    my ($title, $w, $h) = @$conf{qw( title width height )};

    $sdl_app = SDL::App->new(-title  => $title,
                             -width  => $w,
                             -height => $h,
                             -gl     => 1,
                            );
    SDL::ShowCursor(0);
}

At this point, init_conf just defines some configuration properties used immediately in init_window, which contains the first real SDL meat.

init_window performs two important actions. First, it asks SDL::App to create a new window, with the appropriate title, width, and height. The -gl option tells SDL::App to attach an OpenGL 3D-rendering context to this window instead of the default 2D-rendering context. Second, it hides the mouse cursor (while it's within the new window's border) using SDL::ShowCursor(0).

Three Phases of Drawing

Now that I have a nice new window, I'd like do_frame to do something with it. I'll start by breaking the rendering into three phases: prepare, draw, and finish.

sub do_frame
{
    prep_frame();
    draw_frame();
    end_frame();
}

For now, draw_frame contains exactly what do_frame used to contain:

sub draw_frame
{
    print '.';
    sleep 1;
    $done = 1 if $frame == 5;
}

The new code is in prep_frame and end_frame; let's look at prep_frame first:

sub prep_frame
{
    glClear(GL_COLOR_BUFFER_BIT);
}

This is the first actual OpenGL call. Before I explain the details, it's worth pointing out the OpenGL naming conventions. OpenGL's design allows it to work with programming languages that have no concept of namespaces or packages. In order to work around this, all OpenGL routine names look like glFooBar (CamelCase, no underscores, gl prepended), and all OpenGL constant names look like GL_FOO_BAR (UPPERCASE, underscores between words, GL_ prepended). In older languages, this prevents the OpenGL names from colliding with names used in other libraries. In the Perl world, this isn't an issue for object-oriented modules. Because OpenGL is not object-oriented, SDL_Perl takes advantage of this convention and simply imports all of the names into the current package when you write use SDL::OpenGL.

Note: If you read OpenGL code written in C, you may notice a short string of characters appended to the routine names, such as 3fv. This convention differentiates variants that have different numbers of parameters or whose parameters are different types. In Perl, values know their own type and a function's parameters can vary in number, so this is unnecessary. The Perl bindings simply drop these extra characters and SDL::OpenGL do the right thing for you.

The OpenGL call in prep_frame clears the rendering area to black by calling glClear -- the general OpenGL "clear a buffer" routine -- with a constant that indicates it should clear the color buffer. As the name indicates, the color buffer stores the color for each pixel and is what the user sees. Several other OpenGL buffers exist; I'll describe those later.

The alert reader may wonder why the code clears the color buffer to black as opposed to white or some other color. OpenGL relies heavily on the concept of current state. Many OpenGL routines do not actually request any rendering, instead altering one or more variables in the current state so that the next rendering command will perform its action differently. When a program prepares to use OpenGL, which SDL::App::new does for us, the current state is set to (mostly) reasonable defaults. One of these state variables is the color to use when clearing the color buffer. Its default is black, which I haven't bothered to override.

The remaining routine is end_frame :

sub end_frame
{
    $sdl_app->sync;
}

This asks the SDL::App object to synchronize the window contents with those held in OpenGL's color buffer, so that the user can see the rendered image. In this case, it's a black window for five seconds.

Something to See

It's time to draw something in that window. To do so, I need to do three things:

  1. Choose a projection, so that OpenGL knows how I want to look at the scene.
  2. Set the view, so OpenGL knows from which direction to view the scene (the viewpoint) and in which direction I wish to look.
  3. Define an object in the scene, placed where the viewpoint can see it.

To start, I need another config setting, so I'll add another line to the $conf hash in init_conf:

        fovy   => 90,

Next, for my three new functions, I add three new calls at the top of draw_frame:

sub draw_frame
{
    set_projection_3d();
    set_view_3d();
    draw_view();

Choose a Projection

set_projection_3d is as follows:

sub set_projection_3d
{
    my ($fovy, $w, $h) = @$conf{qw( fovy width height )};
    my $aspect = $w / $h;

    glMatrixMode(GL_PROJECTION);
    glLoadIdentity;
    gluPerspective($fovy, $aspect, 1, 1000);

    glMatrixMode(GL_MODELVIEW);
    glLoadIdentity;
}

This is the first place you can see an indication of the hard part of 3D graphics simmering below the surface -- math, and lots of it. 3D-rendering code often includes a fairly hefty load of linear algebra (matrix math, for those blocking out their high school and college years) and trigonometry. Thankfully, OpenGL does a lot of that math under the covers. I've also defined a fairly simple projection and view, so this hides a lot of the complexity for now (aside from some of the OpenGL function names).

The first section of the routine defines the viewing projection. In the simplest case, that means choosing whether to use an orthogonal projection or a perspective projection. Orthogonal projections have no foreshortening. They commonly appear in architectural and engineering drawings, because parts that are the same size also appear the same size, no matter where they are in the scene.

Perspective projections are what we see in the real world with our own eyes or with a camera; distant objects appear smaller than near objects. It's also what you learn in a perspective drawing art class, in which the first assignment is commonly train tracks going off to the horizon. Tracks farther from the viewer appear closer together as does the spacing between the ties. To replicate the real world, I've chosen a perspective projection.

In OpenGL, you not only have to decide between an orthogonal or perspective projection, you have to define its basic dimensions. In other words, how much can you see? For a perspective projection, you define the vertical field of view (FOV), the aspect ratio of the view, and the distance to the nearest and farthest things visible.

The vertical FOV ($fovy in the code) defines the angle from the viewpoint to the lowest and highest visible parts of the scene. If you imagine drawing what someone would see if she were standing with her eyes at the viewpoint, this represents her vertical peripheral vision. If you imagine a camera instead, this depends on the focal length of the lens. A telephoto lens has a very small FOV because the angle from the camera to the top and bottom visible objects is very small. Conversely, a wide-angle lens has a large FOV, and the FOV for a fisheye lens is even larger, approaching 180 degrees.

The aspect ratio comes directly from the dimensions of the drawing area (width/height). This allows OpenGL to compensate for the stretching effect of a non-square window. In this case, the drawing area is square, so the aspect ratio is 1.

After calculating the window's aspect ratio, I tell OpenGL that I want to modify the projection and to start from a blank slate, using glMatrixMode(GL_PROJECTION) and glLoadIdentity. I then call gluPerspective to define the desired perspective. You probably noticed that gluPerspective begins with glu instead of gl, like all of the other calls we've seen. This is because I'm using one of the GLU (OpenGL Utility) routines to cover up some complexity in the equivalent raw OpenGL sequence.

Finally, I switch back to model/view mode, and once again start with a blank slate, using glMatrixMode(GL_MODELVIEW) and glLoadIdentity. You may wonder why I don't include this in the next routine instead of doing it here. I like to make sure routines that change a commonly used OpenGL state, simply as a side effect of their main purpose, return that state to the way they found it, especially if there is no net performance effect to doing so. In this case, I switch temporarily to projection mode and then switch back to the default model/view mode.

Set the View

The next step is to move the viewpoint to somewhere we can see the scene:

sub set_view_3d
{
    # Move the viewpoint so we can see the origin
    glTranslate(0, -2, -10);
}

I'm going to skip the detailed explanation for now, but in short the glTranslate call leaves the viewpoint a few units away from (and above) the origin of the scene, where I'll place my objects. I keep the default viewing direction, because it happens to point right where I want it to.

Define an Object

I'm going to start with a pretty simple scene -- just one object:

sub draw_view
{
    draw_axes();
}

sub draw_axes
{
    # Lines from origin along positive axes, for orientation
    # X axis = red, Y axis = green, Z axis = blue
    glBegin(GL_LINES);
    glColor(1, 0, 0);
    glVertex(0, 0, 0);
    glVertex(1, 0, 0);

    glColor(0, 1, 0);
    glVertex(0, 0, 0);
    glVertex(0, 1, 0);

    glColor(0, 0, 1);
    glVertex(0, 0, 0);
    glVertex(0, 0, 1);
    glEnd;
}

The lone object is itself quite simple, just three short lines extending from the origin along the X, Y, and Z axes. (I'm using "line" in the OpenGL sense, as a line segment, not the infinite line of rigorous mathematics.)

In OpenGL, when you want to define something to render, you must notify OpenGL when you begin and end the definition; these are the glBegin and glEnd calls. In addition, you must tell OpenGL what type of primitive you will use to create your object. There are several types of primitives, including points, lines, and triangles. In addition, each primitive type has variants based on how several primitives in a sequence connect (independently, connected in a strip, and so on). In this case, I use GL_LINES, indicating independently placed line segments.

I want each line to be a different color to make it easier to tell which is which. To set the current drawing color, I call glColor with an RGB (Red, Green, Blue) triplet. In OpenGL, each color component can range from 0 (none) to 1 (full). Therefore, (1, 0, 0) indicates pure red, (0, 1, 0) is pure green, and so on. A medium gray is (.5, .5, .5). For further mnemonic value, I assign the colors so that the RGB triplets match the coordinates of the endpoints of the lines -- red for the X axis, green for Y, and blue for Z.

For each line, after defining the color, I define the endpoints of the line using glVertex. Each line begins at the origin and extends one unit along the appropriate axis. In other words, this sequence defines a red line from (0, 0, 0) to (1, 0, 0):

    glColor(1, 0, 0);
    glVertex(0, 0, 0);
    glVertex(1, 0, 0);

With these routines in place, we finally have something to look at! As you can see, the X axis points to the right, the Y axis points up, and the Z axis points out of the screen toward the viewer (with OpenGL foreshortening it). Note the delay before the object first appears; that's because the sleep at the end of draw_frame creates a pause before end_frame syncs the screen with the drawing area.

Moving Boxes Around

Next, let's try a box. Anyone who's played a First Person Shooter game knows that their worlds have a surplus of boxes (a.k.a. "crates," "storage containers," and so on �- oddly, for storage containers, the larger they are, the less they seem to contain). I'll start with a simple cube and add another call for it to the end of draw_view:

sub draw_view
{
    draw_axes();
    draw_cube();
}

Defining the Cube

Here's the code that actually draws the cube:

sub draw_cube
{
    # A simple cube
    my @indices = qw( 4 5 6 7   1 2 6 5   0 1 5 4
                      0 3 2 1   0 4 7 3   2 3 7 6 );
    my @vertices = ([-1, -1, -1], [ 1, -1, -1],
                    [ 1,  1, -1], [-1,  1, -1],
                    [-1, -1,  1], [ 1, -1,  1],
                    [ 1,  1,  1], [-1,  1,  1]);

    glBegin(GL_QUADS);
    foreach my $face (0 .. 5) {
        foreach my $vertex (0 .. 3) {
            my $index  = $indices[4 * $face + $vertex];
            my $coords = $vertices[$index];
            glVertex(@$coords);
        }
    }
    glEnd;
}

That looks pretty hairy, but it's actually not bad. The @vertices array contains the coordinates for a cube two units on a side, centered at the origin, with its sides aligned with the X, Y, and Z axes. The @indices array defines which four vertices belong to each of the six faces of the cube and in what order to send them to OpenGL. The order is very important; I've arranged it so that, as seen from the outside, the vertices of each face draw in counterclockwise order. Using a consistent order helps OpenGL to determine the front and back side of each polygon; I've chosen to use the default counterclockwise order.

After defining those arrays, I mark the beginning of a series of independent quadrilateral primitives using glBegin(GL_QUADS). I then iterate through each vertex of each face, finding the correct set of coordinates and sending them to OpenGL using glVertex. Finally, I mark the end of this series of primitives using glEnd.

Colloquial Perl purists will no doubt wonder why I have chosen C-style loops (with the attendant index math, yuck), rather than making @indices an array of arrays. Mostly I'm just showing that it's not too hard to deal with this type of input data. When the engine reads object descriptions from files, rather than hand-coded routines, the natural output of the file parser may be flattened. It's often easier to do a little index math than to force the parser to output more structured data (and possibly more efficient too, but that's a clear call for benchmarking).

The result is one blue cube. Why blue? Since I never specified a new color to use, OpenGL went back to the current state and looked up the current drawing color. The last line in the axes was drawn in blue and that's still the current color. Hence one blue cube.

Two Colored Boxes

Let's fix that. At the same time, we can move the new cube out of the way of the axes so we can see them again. Heck, I'll go all out and have two cubes -- one to the left of the axis lines, and one to the right. The nice thing is that because I'm just drawing more of something I've already described, I just need to change draw_view:

sub draw_view
{
    draw_axes();

    glColor(1, 1, 1);
    glTranslate(-2, 0, 0);
    draw_cube();

    glColor(1, 1, 0);
    glTranslate( 2, 0, 0);
    draw_cube();
}

Now I set the current color to white using glColor(1, 1, 1) before drawing the first cube, and to yellow using glColor(1, 1, 0) before drawing the second cube. The glTranslate calls should place the first cube two units to the left (along the negative X axis) and the second cube two units to the right (along the positive X axis).

Cumulative Transformations

Unfortunately, no dice. The white cube is two units to the left, but the yellow cube is right on top of the axis lines again, not two units to the right as intended. This happened because glTranslate calls (and the other transformation calls I'll show later) are cumulative. Unlike routines such as glColor that simply set the current state, most transformation calls instead modify the current state in a certain way. Because of this, the first cube starts at (-2, 0, 0), and the second starts at (-2, 0, 0) + (2, 0, 0) = (0, 0, 0) -- right back at the origin again.

The solution to this problem requires peeking under the covers a little bit. OpenGL transformation calls really just set up a special matrix representing the effect that the requested transformation has on coordinates. OpenGL then multiplies the current matrix by this new transformation matrix and replaces the current matrix with the results of the multiplication.

What I need to fix this problem is some way to save the current matrix before performing a transformation, and then restore it after I'm done with it. Thankfully, OpenGL actually maintains a stack of matrices of each type. I just need to push a copy of the current matrix onto the stack before drawing the white cube, and pop that copy off again afterwards to get back to the state before I did my translation. I'm going to do this for both cubes:

sub draw_view
{
    draw_axes();

    glColor(1, 1, 1);
    glPushMatrix;
    glTranslate(-2, 0, 0);
    draw_cube();
    glPopMatrix;

    glColor(1, 1, 0);
    glPushMatrix;
    glTranslate( 2, 0, 0);
    draw_cube();
    glPopMatrix;
}

That's a bit better. The yellow cube now has its origin at (2, 0, 0), just as intended.

Other Transformations

Earlier I referred to other transformation calls; let's take a look at a few of them. First, I'll scale the boxes (change their size). I'm going to scale the left (white) box uniformly -- in other words, scaling each of its dimensions by the same amount. To show the difference, I'll scale the right (yellow) box non-uniformly, with each dimension scaled differently. Here's the new draw_view:

sub draw_view
{
    draw_axes();

    glColor(1, 1, 1);
    glPushMatrix;
    glTranslate(-4, 0, 0);
    glScale( 2, 2, 2);
    draw_cube();
    glPopMatrix;

    glColor(1, 1, 0);
    glPushMatrix;
    glTranslate( 4, 0, 0);
    glScale(.2, 1, 2);
    draw_cube();
    glPopMatrix;
}

For the white box, I just doubled each dimension; the parameters to glScale are X, Y, and Z multipliers. For the yellow box, I shrunk the X dimension by a factor of 5 (multiplied by .2), left Y alone, and doubled the Z dimension. The boxes are now big enough that I've also pushed them farther apart, hence the updated values for glTranslate that place them four units on either side of the scene origin.

Watch the Rotation

I've done translation and scaling; next up is rotation. To save space here, I'll demonstrate on the yellow cube alone. Here's the new code snippet:

    glColor(1, 1, 0);
    glPushMatrix;
    glRotate( 40, 0, 0, 1);
    glTranslate( 4, 0, 0);
    glScale(.2, 1, 2);
    draw_cube();
    glPopMatrix;

The parameters to glRotate are the number of degrees to rotate and the axis around which to do the rotation. In this case, I chose to rotate 40 degrees around the Z axis (0, 0, 1). The direction of rotation follows the general pattern in OpenGL -- a positive value means counterclockwise when looking down the rotation axis toward the origin.

Order of Transforms

This produces a flying yellow box in the upper-right quadrant. Remember when I said that each new transformation is cumulative? The order matters. To understand why, I like to imagine each transformation as moving, rotating, or scaling the coordinate system in which I draw my objects. In this case, by rotating first, I certainly rotated the box, but I really rotated the entire coordinate system in which I defined the box. This meant the glTranslate call that immediately follows the rotation translated out along a rotated X axis, 40 degrees above the scene's X axis, to be precise.

I'll move the rotation after the other two transformations to fix that:

    glTranslate( 4, 0, 0);
    glScale(.2, 1, 2);
    glRotate( 40, 0, 0, 1);

Now the box isn't flying, but it does appear squashed in an odd way. The problem here is that because the nifty, non-uniform scaling happens before the rotation, I'm now trying to rotate through a space where the dimensions are different sizes. Putting the rotation in the middle fixes it:

    glTranslate( 4, 0, 0);
    glRotate( 40, 0, 0, 1);
    glScale(.2, 1, 2);

If you compare this rendering from this version with the program with no glRotate call, you should see that it does the right thing now.

Whoa, Deep!

The last item I wanted to bring up is what to do when something near the back draws after something near the front. To see what I mean, I'll move the white box so that instead of being four units to the left of the scene origin, it is four units behind it (along the negative Z axis). That merely involves changing the white box's glTranslate call from this:

    glTranslate(-4, 0, 0);

to this:

    glTranslate( 0, 0, -4);

As you can see, even though the white box should appear behind the axis lines, it instead appears in front because OpenGL drew it after the axis lines. By default, OpenGL assumes you intended to do this (it is more efficient to make this assumption), but I didn't. To fix this, I need to tell OpenGL to pay attention to the depth of the various objects in the scene and not to overwrite near objects with far ones.

To do this, I need to enable OpenGL's depth buffer. This is similar to the color buffer, which stores the color of every pixel drawn. Instead of storing the color, however, it stores the depth (distance from the viewpoint along the viewing direction) of every pixel. Just like the color buffer, I need to clear the depth buffer each frame. Instead of clearing it to black, OpenGL clears it to the maximum depth value, so that any later rendering within the visible scene will be closer.

I also need to tell OpenGL that it should perform a test each time it wants to draw a pixel, comparing the depth of the new pixel with what's already in the depth buffer. If the new pixel is farther from the viewer than the pixel it is about to replace, it's safe to ignore the new pixel and to leave alone the old color. Here's the updated prep_frame:

sub prep_frame
{
    glClear(GL_COLOR_BUFFER_BIT |
            GL_DEPTH_BUFFER_BIT );

    glEnable(GL_DEPTH_TEST);
}

In this version, I tell glClear to clear both the color buffer and the depth buffer. You can now see why the constant names end with _BIT; they are, in fact, bit masks. The reason for this odd interface is purely efficiency -- some OpenGL implementations can very rapidly clear all requested buffers simultaneously, and making the request for all needed buffers in just one call allows this optimization. As for the choice of bit mask rather than a list of constants, SDL_Perl reflects the underlying C interface, so that people comfortable with that can more easily cross over to using OpenGL under Perl.

The second routine I call, glEnable, is actually one of the most commonly used OpenGL routines, despite the fact that this is the first we've seen of it. Much of the OpenGL current state is a set of flags that tell OpenGL when to do (or not do) certain things. glEnable and the corresponding glDisable set these flags as desired. In this case, I turn on the flag that tells OpenGL to perform the depth test, throwing away pixels drawn in the wrong order.

With these changes, we can now once again see the axis lines, this time in front of the white box where they belong.

Conclusion

The final results may look simple, but we've come a long way. I started with some basic boilerplate and a simple main loop. I didn't even load SDL or OpenGL or open a window. By the end, I'd added a window to draw on; projection and viewing setup; multiple objects of different types, built using different OpenGL primitives, drawn in different colors, and transformed several different ways; and correct handling of out-of-order drawing.

That's a lot for now, but we're just starting. Next time I'll cover moving the viewpoint, SDL keyboard handling, and compensating for frame rate variations. I'll build on the example source code built in this article, so feel free to download it and use it for your own applications.

In the meantime, if you'd like to learn more visit the OpenGL and SDL websites; each contains (and links to) mountains of information.

Visit the home of the Perl programming language: Perl.org

Sponsored by

Monthly Archives

Powered by Movable Type 5.13-en