August 2010 Archives

People of Perl 6: Carl Mäsak

Carl Mäsak is a developer of Rakudo Perl 6 and arguably the most dedicated bug wrangler. He's contributed to more bug reports than anyone else. If you find that Rakudo does not do as you expect, you'll likely talk to him on #perl6 for help triaging and categorizing your bug.

In his own words, here's how he's helped make Perl 6 real.

What's your background?

I spent my teens learning programming by writing hundreds of small graphical games in BASIC. A combination of university studies and employment has given me a more solid foundation in programming, but I'm still catching up on the theory in many places.

What's your primary interest in Perl 6?

Helping bring it from the world of ideas into the world of implementations. As part of that, helping build up all the things around the implementations: libraries, tooling, documentation, culture. Making Perl 6 practically usable.

When did you start contributing to Perl 6?

I got pulled in during the early Pugs days in 2005, but only staying on the outskirts of the community. I saw with regret how the Pugs development slowed and stopped during 2007, and with growing excitement how the Rakudo development got going in 2008. My real entrance as a contributor was that summer, when I secretly co-wrote a wiki engine on top of Rakudo.

What have you worked on?

A wiki engine (November), a 3-d connection game (Druid), a Perl 6 project installer (proto), a set of web development modules (Web.pm), a grammar engine (GGE), and a dozen smaller modules and projects.

I also occasionally contribute commits to Rakudo, to the Perl 6 specification, and I regularly blog about the progress and culture of Perl 6. My biggest single contribution is probably submitting hundreds of Rakudo bugs that I or others have found in the course of using Rakudo Perl 6.

What feature was your moment of epiphany with Perl 6?

I just remember being awed by the visionary tone and promise of the Apocalypses as they were rolled out. I no longer have any memory of like one particular feature more than the others. Since I program more in Perl 6 than in any other language nowadays, I tend to take them for granted. :)

What feature of Perl 6 will (and should) other languages steal?

I think the new regex/grammar features will be so attractive that other languages won't be able to keep their grubby hands off them. Of course, they also won't get the pervasiveness of regexes and grammars just by tacking them onto an existing language.

What has surprised you about the process of design?

Primarily the extremely low amounts of vandalism, considering that hundreds of people have write access to the documents which specify the language itself. Accidental damage is also often quickly corrected, and more subtle errors get discovered and corrected in a wiki-like manner in the long term.

I've also gained a new respect for what a "holistic" process the design of a language such as Perl 6 can be sometimes. Whether some feature turns out to be a good idea is determined by dozens of minute interactions in the spec, not all of them "local", and some of them outright emergent.

How did you learn the language?

The hard way. :-) By trying to express every new thought I have and seeing what fails: Perl 6, Rakudo, or my thought. I'm still learning.

Where does an interested novice start to learn the language?

Come to #perl6. Familiarize yourself with perl6.org. Get a pet project. Experiment. Have fun.

How do you make a language intended to last for 20 years?

I'm not qualified to fully answer that. The one piece of the puzzle I do have is that some pieces of software out there are stale and dead, whereas others are limber, extensible and moving. It's important to design for the eventuality that the user of the language knows best.

What makes a feature or a technique "Perlish"?

A number of small, sometimes contradictory criteria. It gets the job done. It's down-to-earth rather than abstraction-laden. It's practical rather than orthogonal. It's often consistent in strange and unexpected ways. It favours you rather than the compiler implementor. It goes an extra mile to be user friendly rather than cryptic. It doesn't oversimplify. It encapsulates some linguistic notion. It scales with your needs. It's as much about language culture as it is about language implementation.

What easy things are easier and which harder things are more possible now? Why?

Sub/method signatures, the type system, grammars, extending the language, the metamodel... all of these make things that were possible but tricky or ugly before downright easy or beautiful. Perl 5 can do a whole lot of the above things using modules, but with Perl 6 you get them out-of-the-box, and fully integrated.

What feature do you most await before you use Perl 6 for your own serious projects?

I'm looking forward to more speed and stability in the existing implementations, mostly in Rakudo. I think excellent database and web support will create great changes in the way Perl 6 is used and presented.

As for actual features, I'm excited that Synopsis 9 might be mostly implemented this year—it contains a lot of exciting syntactic sugar.

Also very much looking forward to playing with macros—I've been waiting five years for them now! :-)

What has surprised you about the process of development?

I'm delighted that what drives Rakudo development and makes it efficient is the fact that the group of people organized around it are able to communicate their needs and abilities, able to delegate as well as do large chunks of work in isolation as called for by the situation. I'm thinking of Patrick when I say that, but I see those traits in others as well.

It also doesn't hurt that those of us involved in development form an increasingly knit-together group of people who enjoy each other's company, not only on IRC but away from the keyboard as well.

What does Rakudo need for wider deployment?

In the time before it gets the obvious the-more-the-better features— stability and speed—what Rakudo needs most is people who are brave enough to deploy it in new and interesting situations. It is still the case that when we try new things with Rakudo, we discover new bugs and corner cases, and as a result the whole development process benefits. So we need people who want to break new ground.

What comes next after Rakudo Star?

If you mean the name, there's no consensus yet. Suggestions welcome. All we agree on is that Rakudo Nova might not fly, Rakudo Neutron Star sounds a bit heavy, and Rakudo Black Hole would kinda suck.

As for what features come next after Rakudo Star, I think an important part of the development after the Rakudo Star release will be to get feedback from people who use it, in order to know better what to focus on next. I know the core devs have some ideas, but there's also room for course corrections. Whether the next distribution release will be after half a year, a full year, or some other timespan, depends a lot on that too.

What feature do you most look forward to in a future version of Perl 6?

A future version of Perl 6 the specification? The mind boggles. I think I don't have any further demands on the specification than what we already have in there. I'm fully occupied trying to think up ways to abuse the features from the current spec as they come online.

Larry wanted the community to rewrite itself just as it redesigned and implemented the language. How have you seen that process work?

People seem to agree that the #perl6 channel on freenode is a generally friendly place. It's at least partly the result of conscious effort. On the perl6-* emailing lists you will at times see the most informative and friendly RTFM emails you've ever read.

What's still an open question for me is how these community features will scale, as the ratio of newbies to regulars skyrockets, as the frequently asked questions become more frequent, and as Perl 6 enters an "Eternal September" phase.

How do you keep up with spec changes?

I tend to get involved in the discussion about them. :-)

What one thing would you change about the implementation history so far?

It's clear in retrospect that we should have started both Rakudo and Pugs in 2001, not years later. Of course, we had neither the knowledge we do today, nor the people.

How can people help you?

By being curious about Perl 6, by writing their first little script, by finding a module (new or extant) to work on, by reporting bugs or proposing enchancements in Rakudo or the various projects, by bringing interesting discussions to the channel, by interacting with the community, and by being nice.

What misconceptions do people have about the project that need addressing?

Most misconceptions seem to me to be merely the result of a deplorable lack of correct information, mixed with blindly cargo-culted mockery.

Some people seem to think that working on a language design for ten years, adapting it both to new ideas and to the feedback from implementations, in itself qualifies as a failure of some sort. I see a language growing both more real and more realistic every day.

Some people haven't been reached by the news that we actually have runnable implementations of Perl 6, and have had so for the past five years. Those people usually become very happy to hear that we do.

Others consider the Perl 6 effort as "merely an academic effort", whose purpose will in the end only be to inform Perl 5 in certain ways, and whose ivory-tower denizens will never truly reconnect with reality. This is the only misconception that I, having written dozens of Perl 6 modules which actually run, and having as my focus making Perl 6 more practically usable, can sometimes feel saddened by.

What projects are missing in the world of Perl 6?

Nearly all of them.

Carl and other Perl 6 developers are hard at work on both the Perl 6 specification as well as the Rakudo Perl 6 implementation. Rakudo Star 2010.08 is now available, with better performance, fewer bugs, and more features.

In 1999 and 2000 I worked on the Everything Engine, a website management system written in Perl 5 which runs sites like Everything 2 and PerlMonks. The system has its flaws--many of them due to our inexperience building and maintaining large systems with the Perl of the time.

We could build it much better today, with modern libraries and techniques in Perl. In truth, building such a system today would solve a lot of uses for MVC applications.

How Everything Works

The fundamental concept of Everything is that everything is a node. A node is an individual entity containing attributes, including a unique identifier and one or more addressing mechanisms expressible through a URI. Each node also has a nodetype (itself a node) which contains information about the node's characteristics, such as the attributes it contains, the types of operations it supports, and any mechanisms to view node information.

In other words, a node is an object, a nodetype is a class, and there's a metaobject protocol governing the relationships of nodes and nodetypes.

The Everything Engine provides a sort of routing system which might remind you of routes in one of the new heavy-model MVC systems. In one sense, Everything provided its own implementation of MVC, where URIs and URI components helped the controller find the appropriate model (node) and model operation (nodemethod), which eventually caused the rendering of a series of templates, or the view.

The system provides node inheritance for attributes and behaviors such as creating, reading, updating, and deleting nodes. Because every node inherited from node, every node has at least general behavior through any interface into the system. If you create a new nodetype, you can do useful things with nodes of that type without having to write any specific views or mess with routing or the like. You have the option of making prettier or more featureful or better or at least different views, but the default inherited views and behaviors are sufficient for the basic CRUD operations.

What Everything Didn't and Did Well

Not everything went right. Remember, this system came about in 1998 and 1999, just as mod_perl was the shiny new technology in the world of Perl web development. Object-relational mapping was still difficult (if even understood much), CGI.pm was still the king, and Template Toolkit hadn't taken over as the obvious first place to look for your templating needs.

Our hand-written ORM has its flaws, as dealing with node inheritance and lookup of node types and (yes) serializing code to the database to eval into a running instance of the system at initialization time. The XML serialization scheme for bundling core nodes and custom nodes was even worse, not only due to our use of the DOM.

Without a well-understood mechanism and framework and example of doing MVC well in Perl 5, Everything's custom interpretation of MVC was odd. Where MVC helps separate application responsibilities into loosely-coupled layers, passing around a CGI object to models and templates violates that encapsulation. (The less said about the custom method dispatch strategy the better.)

Most of the problems with the system are obvious in retrospect, over a decade later (especially with a decade of experience creating new systems, maintaining existing systems, and stealing ideas from other projects which have made different mistakes).

Even with all of those mistakes, the system worked reasonably well in many cases. Adding new features was easy (even if deploying them is less so). Creating new behaviors by reusing existing primitives makes building simple systems easy. The usability provided by inherited defaults made it easy to iterate and experiment and refine new behaviors.

We also eventually produced a system to bind UI widgets--HTML, in our case--to node attributes. That made displaying and editing mechanisms much, much easier.

I wouldn't start a new site with Everything as it exists today, but I've wanted a modern version of it for a long, long time. Modern Perl has almost all of the pieces needed to rebuild it.

Doing Everything Better

The Moose object system provides an obvious way to define nodes. A node is merely a Moose object, and a nodetype is its class. This provides a well-understood and robust mechanism for dealing with attributes and methods and roles and the like, and the metaobject protocol provided by Class::MOP allows introspective capabilities which will become obviously important soon.

Plack support is obviously the current best way to deploy a web application in Perl 5, as it allows reusable middleware and offers many deployment strategies.

DBIx::Class is the first ORM to consider in Perl 5 right now. I'm partial to KiokuDB for applications where I need persistent objects but don't need to provide a relational interface to interact with the data. The ability to use either one as a serialization backend is important.

Any of the modern frameworks or libraries or toolkits for providing the controller part of the MVC application will do. In particular, all this layer of the application needs to do is to map requests to nodes, manage concerns of user authentication and logging, invoke the fat model objects to perform their business actions, then dispatch to the view to render the appropriate information in the user interface. I like the simplicity of Dancer, but anything compatible with Plack will work for web applications.

A New Architecture for Perl Apps

Here's my vision.

I run a publishing company. I want to manage a web site with information about existing and upcoming books.

I start by defining some models: a Book has a title, an ISBN, a price, one or more Authors, a short blurb, a full description, and publication date, and a cover image. An Author has a name, a biography, and an image.

For each attribute of each model, I choose two types: the type for the object attribute itself (an ISBN is a ten- or thirteen-digit number, an author is an Author object, et cetera) as well as the type for the UI (Authors display as a list of authors and they require a multiple selection UI widget to display). That author selection widget is interesting because it can be a parametric role: a role which knows how to display multiple somethings, where that something is a parameter in this case constrained only to Authors.

Obviously I need some mechanism to define new UI and attribute types, but assume that the system comes with a rich set of types (password, phone number, long input, short input, et cetera) from which I can build more.

My serialization layer already knows how to serialize these models. That's part of the magic of using KiokuDB, and there are mechanisms available for DBIx::Class to perform the same mapping.

Given these models, the controller layer can create default URI mapping routes for basic CRUD operations. The HTTP verbs PUT, GET, POST, and DELETE map nicely. As well, the first URI component beneath the application itself can map to the type of model required, such that http://example.com/awesomebooksaboutperl/authors/chromatic obviously returns my author page. Without a unique identifier, http://example.com/awesomebooksaboutperl/authors/ could list all authors.

With the UI information associated with models, I don't even have to write any templates to get a bare-bones UI. The system can use Moose's introspection mechanism to read all of the object attributes then bind them to HTML widgets for the appropriate display type (reading, creating, and updating). This is particularly easy with a system like Moose where another parametric role can customize the appropriate elements to render based on access controls. That is, users can update their own passwords and administrators can update anyone's passwords, but users cannot even see the password entry fields for other users. The model object decorated with this UI role can decline to make inappropriate information available to the template rendering system at all.

Even better, the UI decoration role can provide different destination output types as well, such as JSON or XML or even serialized objects themselves suitable for transportation between similar systems.

I care most about HTML for this web application, but it's nice to have a JSON or Atom view of my data, especially because I can define another route (or perhaps you get this by adding an attribute to your models) which generates syndication information automatically; if I add a new book, it's part of the http://example.com/awesomeperlbooks/comingsoon.xml Atom feed. Internally, the controller might map that URI to http://example.com/awesomeperlbooks/books/?view=atom;sort_by=desc;limit=10.

Whatever the other output options, I want my HTML to make generous use of CSS selectors, such that I have the option of customizing the result purely in CSS, where possible. (I don't mind writing custom HTML, but the less of that I have to do the better.) This is because it's possible to build a big page out of several models rendered together: each model should be able to render as a fragment, which makes Ajax applications easier to write.

Perhaps the real benefit of this system is that it can host itself, in that it's very easy to write an administrative console which allows people to define their own models and types and widgets using the system as its own interface. I'm still likely to write my own models by hand in Vim, but I don't mind customizing an HTML template fragment in a web browser window on a development server, at least if I can deploy the entire system as if I'd written it by hand to files in the filesystem.

I've discussed this system with a few people and various projects exist to make much of it work. Consider modeling, serialization, and deployment solved thanks to Moose, DBIx::Class and KiokuDB, and Plack. What's left is some middleware, some conventions for routing and mapping, and a little bit of magic UI widget binding and default templates. I wish we'd had this great technology a decade ago, but now with modern Perl we may be able to create a postmodern fat-model MVC web framework that makes the easy things trivial, the hard things easy, and the annoying things wonderful.

People of Perl 6: Jonathan Worthington

Jonathan Worthington is one of the lead developers of Rakudo Perl 6. A couple of months before the Rakudo Star release, Perl.com interviewed him about his work on Perl 6 and what the rest of us should expect as we explore this new language. Follow Jonathan's work at 6guts.

What's your background?

I discovered programming when I was about eight years old, and got hooked. It started out with Logo and BBC Micro BASIC on some computers at school. Several years and a couple of PC-based dialects of Basic later, my family got internet access at home, and a couple of years later I ran across Perl.

I ended up going off to university to do Computer Science. I questioned it a bit—I wasn't certain my childhood hobby would still be interesting as a topic of study and a future career. Happily, it deepened my curiosity rather than bored me. I took a particular interest in languages, compilers and type systems, though the natural language processing courses kind of caught my interest too.

That all came to an end several years back, when I graduated. Since then, I've done a few years freelancing and living abroad, and recently joined a startup company in Sweden. Amusingly, it makes the third country starting with an "S" that I've lived in since graduating—the other two were sunny Spain and the lesser-known but lovely Slovakia, a place I remain very fond of.

What's your primary interest in Perl 6?

At the moment, I'm primarily interested in writing a good compiler for it. Beyond that, I'd like to use it to build more cool things, though I've little idea what yet. We'll see what comes along.

When did you start contributing to Perl 6?

Back in 2003, a couple of years after discovering Perl, I heard about Perl 6. I found the apocalypses engaging, and joined a couple of mailing lists. A lot of perl6-language bewildered me. Well, some things never change. :-) I felt way out of my depth on the language design front back then, but then found the Parrot project, which was a little more concrete. I wrote a couple of patches to improve Windows support.

I did more bits with Parrot over the following years—mostly on the bytecode format and some object orientation bits. I didn't really get involved with the Perl 6 compiler itself until 2007. That summer, I went to OSCON, drank a few beers and then told Larry Wall, of all people, that implementing junctions in the Perl 6 on Parrot compiler sounded interesting.

Sadly, autumn of 2007 saw me pretty depressed. I withdrew from hacking on just about everything, went to wander around China for a month and hurriedly arranged to move abroad, in hope of putting some distance between myself and things. With the Spanish sun to improve my mood, and noticing that the latest iteration of the Perl 6 compiler—built on the Parrot Compiler Toolkit—had landed, it felt like time to try and make good on my junctions remark.

What have you worked on?

Well, not long after digging in to junctions, I discovered that doing junctions properly meant doing multiple dispatch... and doing multiple dispatch meant doing a bunch of the type system stuff... and doing that meant being able to declare classes and roles. A couple of years of hacking later, and with plenty of input and help from others, we're doing pretty well on all of those areas now in Rakudo. :-)

What feature was your moment of epiphany with Perl 6?

That's a hard one. There are many interesting features in Perl 6—some of them big things, some of them much smaller. What I've really come to admire isn't so much the features themselves, but rather the much smaller number of fundamentals that they're all built on top of, and how they them form a coherent whole that is greater than its parts.

What feature of Perl 6 will (and should) other languages steal?

Perl has long been a leader in regexes, and in many senses Perl 6 grammars and its new regex syntax are a game changer. Parsing problems that are beyond traditional regexes are now often easily expressible. Additionally, the changes in Perl 6 seriously try to address the cultural problems; good software development practices, such as code re-use, should also apply to regexes, and that—amongst other things—is now made easy.

What has surprised you about the process of design?

The willingness to revisit things when an implementation points out issues, and an overriding commitment to get it right, rather than just get it out the door in whatever shape as soon as possible. While it is somewhat frustrating for those waiting to use Perl 6 in production, and to some degree for those of us implementing it too when things get re-hashed, I'm also convinced that the Perl 6 we deliver will be better for the time that's been taken over it.

How did you learn the language?

By implementing a chunk of it. :-)

Where does an interested novice start to learn the language?

Drop along to perl6.org for links to the latest and greatest in terms of documentation and tutorials. Some of us are also working on a book (Using Perl 6). And of course, don't forget to join the #perl6 IRC channel. It's a friendly place, and a great place to get questions answered.

How do you make a language intended to last for 20 years?

While 20 years sounds a long time, in many senses if a language gets a reasonable level of adoption—which I do hope Perl 6 will—it's easy enough for legacy code to still be in production 20 years on.

The more interesting challenge is how to make a language that can stay at the forefront for 20 years and still be considered modern. Since what's considered modern will of course mutate, that means the language has to be able to be designed with the ability to mutate too. Handling language mutations sanely, and making sure it's always clear what "dialect" of Perl 6 is being spoken, has been one of the big challenges in making Perl 6.

What makes a feature or a technique "Perlish"?

It makes an easy thing easy, or a hard thing possible. The amount of code is proportional to the task at hand. It feels natural. It solves a real, practical problem, so you can get your work done efficiently and have time to go for a beer.

What easy things are easier and which harder things are more possible now? Why?

I'd say writing OO code is decidedly easier now, particularly for those coming from other non-Perl language backgrounds. At the same time, so is writing functional code. And writing clean code is most certainly easier. Language extensions are an example of a harder thing that is made much more possible in Perl 6; introducing a new operator isn't that much harder than writing a subroutine, you just have to give it an interesting looking name.

What feature are you most awaiting before you use Perl 6 for your own serious projects?

In a slightly odd sense, I am using Perl 6 for a serious project; big chunks of Rakudo are actually written in Perl 6. That aside, though, feature wise I think Rakudo is doing pretty well; the things I'm most waiting on—and helping us improve on—are issues like performance, stability and having a good range of modules to draw on. If I had to identify the next big features we need, though, it's concurrency support and native type support.

What has surprised you about the process of development?

The high standards at which it is conducted. For example, we have a lot of effort going in to testing, and test coverage is taken seriously and highly valued by all all of those working on Rakudo. People often code-review each other's patches. Discussion on the channel and on the mailing lists—even in the face of opposing views—is just about always polite and friendly. The Perl 6 team is easily the best I've ever had the privilege to work with.

What does Rakudo need for wider deployment?

Wider deployment means growing the users base. People should choose a technology primarily on the merits of the technology itself. Therefore, to grow a user base, Rakudo needs to deliver not only the benefits of the Perl 6 language itself, but also a stable and performant implementation of it. It's a wider issue than Rakudo, but we also need lots of good modules and, perhaps, some kind of killer app. Those kinds of things will come from the community at large rather than just the core Rakudo team, however they matter significantly to Rakudo's own success.

What comes next after Rakudo Star?

I often talk about Rakudo development as being like going for a hike up a mountain. While the most impressive view will be at that top—when we have a complete Perl 6 implementation—at some points along the way there will be good views, and it's good to pause and enjoy them. Rakudo is one of those points on the journey—in fact, the most significant so far.

Rakudo has caused us to focus for a while on trying to get something useful and usable. That has been valuable, however there are still some big items to take care of on the way to the mountain peak. Those will once again take center stage after Rakudo *.

Two big areas of focus will be on native type handling and parallelism. Part of the native type handling will involve a re-visit of how objects look internally. Part of this is being able to store native types in them, not just other objects. Additionally, we can actually do the vast majority of object attribute lookups as index offsets instead of hash lookups, which should be a big performance win. Lexical variable access can also be optimized in a similar fashion. We have a lot of statically known information about a Perl 6 program that we pretty much just throw away today.

We will also be transforming Rakudo from a compiler with one backend—Parrot—into a compiler with multiple backends. We've had an architecture open to such things for some time now, but getting the core features in place and actually delivering something of note have been far more important goals so far.

However, at this point, I think some things—especially parallelism, an area where Parrot is currently weak—will be much easier to do an initial implementation of for another backend. It usually takes at least a first cut implementation and some iteration to get the specification solidified and refined, and it will be easier to do that on a base that offers solid concurrency primitives. I also hope that having other backends will help us grow both the user base and the developer base.

What feature do you most look forward to in a future version of Perl 6?

I'm really looking forward to having good threading and parallel programming support—but perhaps that's mostly because we'll be done with the hard work of making it happen!

Larry wanted the community to rewrite itself just as it redesigned and implemented the language. How have you seen that process work?

The state of the Perl 6 community today is something that is dear to many of us who are a part of it. People frequently comment how the #perl6 IRC channel is somehow different to the norm; it's very open to newcomers and beginners, and people tend to have a very slow burning fuse. The challenge, of course, is scaling that community without losing all of the nice things about it. Significantly, the key players all really want to work out how to do that.

How do you keep up with spec changes?

Mostly through a very regular presence on #perl6, where a lot of them are thrashed out. I also read the spec change commits as they are reported on the perl6-language list.

What one thing would you change about the implementation history so far?

Implementation having been started sooner. Implementations have been the primary driver towards the spec converging on something implementable and sane. That said, the pre-implementation, tossing-about-ideas phase was, of course, important.

How can people help you?

There's so much that needs doing in the Perl 6 world today! Generally, I just advise interested people to take a look around, spot something interesting to help with or something that's missing that they would like to work on, and dig in! Perl 6 is very much about -Ofun. For some of us, fun is compiler guts. For others, fun is web frameworks. For others, fun is making awesome documentation. If you want to help, find what's fun for you, do it well, and make your mark on the evolving Perl 6 community.

What misconceptions do people have about the project that need addressing?

Perhaps the biggest one is that some people equate "taking a long time" with "not going to happen". It's an easy enough misconception to form given the project's long time scale, especially for anyone not really following the day-to-day progress. Of course, it's a hard misconception to understand for those who are following it too. :-)

What projects are missing in the world of Perl 6?

Of course, there's thousands of modules to be written for all kinds of needs from popular to niche. One area that certainly needs more effort, however, is documentation. It's not like there aren't projects started, more just too few hands for the amount of work that needs to be done.

Visit the home of the Perl programming language: Perl.org

Sponsored by

Monthly Archives

Powered by Movable Type 5.13-en