Recently in Community Category

Inside YAPC::NA 2005

The Perl Foundation organizes and holds several community-based Perl conferences each year. This year's North American conference, YAPC::NA 2005 is in Toronto, Canada, June 27-29. chromatic recently interviewed Richard Dice, organizer of the conference this year, to discuss his plans and experiences.

Can you tell our readers a little bit about yourself?

Richard Dice: Richard's just this guy, see?

I'm 32, born in Montreal and grew up in the sleepy southern Ontario city of London, Ontario. I did my undergrad degree in astronomy and applied mathematics at the University of Western Ontario. I moved to The Big Smoke (Toronto) in 1996 and have lived here (with brief stints in Montreal again and in Waterloo, ON) since then. I finished my MBA at the University of Toronto about a year ago, and I have recently started a job as a senior IT consultant at a company in Toronto called Information Balance. I've been married for not quite 2 years now, and we're soon to be moving into our first house. (The day after YAPC! Ugh!)

I've been a Unix-head since 1992 and a Linux guy since 1994, which is around the same time that I started programming in Perl, coming to it from C. (The first Perl program I ever wrote took 45 minutes. To do the same in C would have taken me three days. I never looked back.)

I have a love of good beer and scotch (and know everywhere in Toronto to get both), overestimate my ability to speak French, have an ever-growing personal library that my wife laments, and I go to all the good raves in the city. (I get in to most of the after-parties, too.)

Also, I'm a cat person.

How did you decide that not only did you want someone to host a YAPC in Toronto, but that you were the one to write the proposal and organize things?

RD: I blame Damian.

Damian Conway is a professor of computer science at Monash University in Melbourne, Australia, a Perl programmer par excellence, and an amazing public speaker. In the latter half of 2000, Yet Another Society (also known as The Perl Foundation) raised money to "buy" Damian for the year 2001, releasing him from his professorial duties at Monash and tasking him in part to tour the world speaking about Perl. (The Perl 6 announcement had just recently been made at the time and it was thought that he would make an excellent "champion" of the cause.)

In early 2001, I was in the process of moving from Montreal to Toronto when I noticed that there was a small gap in Damian's tour schedule in June. I had seen him the year before at YAPC 19100 in Pittsburgh and thought he was simply amazing. I wanted to support his mission to evangelize Perl. So I wrote him an email, introduced myself, and asked him to come to Toronto to present a few free public talks on Perl. I didn't want for him to say no, so in order to help make the decision easier I told him I'd make it all expenses paid. He quickly replied to say yes. Then, I wrote the Toronto Perl Monger (TPM) mailing list (which had never heard of me before this) and asked for help with fundraising to bring Damian to Toronto. It took some doing on my part, but in the end Damian's trip was completely covered and I didn't have to back-stop it. Not only was the trip a success financially, but everyone really seemed to enjoy the experience. This was my introduction to the Toronto Perl Mongers, to Damian, and to the wide world of Perl community organization.

Just before he left Toronto, Damian suggested that I consider organizing the same for him during his North American summer tour in 2002. How could I say no? Only this time, with a year to plan things, I could turn it from a small event into Damian-pooloza. Spread out over a week in June 2002 he presented three talks, each with around 250 people in attendance. The budget was also an order of magnitude greater than what I raised the year before. That was when some people locally (and Damian) started suggesting to me that I organize a YAPC in Toronto. I cringed, because I had known the suggestion was going to be raised eventually, and because I knew how much work it would be. But I also knew that, again, there was no way I could say no.

In the summer of 2003 I organized TPM to put together a bid for YAPC::NA 2004, which lost to the Buffalo bid (as judged by The Perl Foundation). And I'm glad that it did! The Buffalo team did a great job, and it just spurred us on to try even harder for our 2005 bid, which we won.

You offered to fund Damian's trip to Toronto without having things lined up first?

RD: That's correct. People are often surprised when I mention this aspect of things.

Is that confidence or something else?

RD: Some of it is confidence, but only part of it would be confidence in myself. (Though certainly I've been accused of this more than once.) The other part is confidence in the Perl community, its generosity and its ability to recognize when something good (e.g. a Damian Conway series of talks) is about to happen.

When I first hatched the scheme to bring Damian to Toronto in my head, I realized that it had all the makings for an organizational race condition. That is, if I went to Damian telling him that we would like him to come to Toronto conditional on funding being found, then he'd say "Great! Get back to me when you've got the funding and I'll make my plans." And then I'd turn to the Toronto Perl community to ask for funding to bring Damian to Toronto, and the reply I'd get would be "I'd love to donate! Tell me when Damian is confirmed and I'll contribute something." That's the kind of situation that just eats planning time away until it's all gone. To break this vicious cycle before it even started I decided to guarantee the funds myself. It worked. There were risks, but the potential rewards were too great not to make the effort.

I remember Damian lobbying for the Toronto bid for 2004. Obviously your approach worked.

RD: I assure you that no kick-back schemes were involved. :-)

I think that whatever support Damian showed for Toronto would have come from his experience in seeing us "pull out all the stops" when we invited him to Toronto in the past. I guess he had confidence in us to do the same, but for hundreds of people this time and not just him.

What's the process of putting together the budget like? I'm not looking for specific details and secrets, just some idea of what goes into organizing a conference.

RD: The budget is everything. The Perl Foundation isn't some huge organization with bottomless pits of money--YAPC has to stand on its own, financially at least. I never allowed myself to plan a budget where I wasn't 99.9% sure that no losses would occur. On the other hand, I didn't want to have a conference that was so skimpy on goodies that people would be talking to each other years from now saying things like, "Remember YAPC in Toronto? That just sucked!"

This made for a rather iterative process. The conference budget was designed "modularly", so as I identified more funding I could add on to the conference. The original budgetary pass at the conference did look pretty sucky. It wouldn't lose any money, but it would only barely make the YAPC grade, too. So then the effort shifted over to finding corporate sponsorships. This was a lot of hard work, but it has been entirely worthwhile. We have had a lot of excellent people and companies support the conference well beyond the call of duty. (Please visit the sponsors page on the YAPC::NA web site to see what I mean and who I'm talking about.)

The other main point regarding the budgetary process is that I didn't do it alone. Eric Bower, one of the Toronto Mongers, stepped forward to create the budgetary spreadsheet models that he and I consulted (and still consult) regularly. The latest iteration of this process has so many smarts built in to it that I wonder sometimes if I could just step away from planning the conference from here on in and let the spreadsheet take care of things by itself. :-)

In the past couple of years, I've heard that there's more collaboration between YAPC organizers than before. (Jeff Bisbee talked about this quite a bit after the YAPC::2003 in Florida.) Is there a community of burned-out ex-organizers that still work together?

RD: The first rule of YAPC ex-organizer club is you do not talk about YAPC ex-organizer club.

Seeing as how I just broke the first rule, I may as well spill the beans entirely.

The majority of the contact I have with the ex-organizers is with Kevin Meltzer (YAPC:NA 2003 in Boca Raton, FL, alongside Jeff Bisbee) and Jim Brandt (YAPC::NA 2004 in Buffalo). Jeff and Kevin glommed themselves onto The Perl Foundation to act as the core of the Conferences Committee, and Jim is just a generally accessible nice guy who can't hide from me, as I've got him on AIM. I've had a few emails with Sarah Burcham (YAPC::NA 2002 in St. Louis) and Rich Lafferty (YAPC::NA 2001 in Montreal) but nothing major.

In addition to the YAPC organizers of previous years, I have been in a great deal of contact with Allison Randal, president of The Perl Foundation, and Kurt DeMaagd, treasurer and general "boy Friday" of The Perl Foundation. He's the unsung hero in all of this as far as I can tell.

Truth be told, even if the previous YAPC organizers were incredibly tight and available to me, I'm not sure what I would be able to do with the "wisdom of the past" they could offer. They might be able to give me a sense of what needs to be accomplished and by what T-minus-YAPC date, and maybe they could give me some leads in terms of corporate sponsorship, but these things really aren't the bottleneck. The early bottleneck for the conference this year has been money and making a budget for the conference that works. Doing that took so much time that things we would have liked to do sooner had to wait too long for my liking. (For example, opening up attendee registration.) The current bottleneck is finding people who can look after task implementation details on the ground here in Toronto. For that, I work with my local TPM members.

How does organizing a conference compare to organizing a software project?

RD: That's a really interesting question. I hadn't considered this before, but I can posit a few possibilities.

First, I want to make a distinction between types of software projects: commercial and open source. I think these two types have quite different organizational characteristics. Organizing a YAPC is more like organizing an open source software project, but it has elements of commercial organization, too.

The open source software aspect of YAPC organizing is that everyone involved is a volunteer. I can only get people to move in the needed direction by setting an example, moral suasion, persuasion, and by being the obvious best person to make decisions and lead the group. That said, I can't make anybody do anything they don't want to. Even if I was ostensibly "successful" in talking someone into something they were reluctant to do, the end result is that they wouldn't put in enough time and wouldn't do a good enough job. I have to be very sensitive to finding the right people to do the right things.

Fortunately, it's not like I'm trying to make an organization that sells widgets. I'm running a YAPC, which is an intrinsically interesting and exciting thing. There are lots of people in TPM who want to help out and are willing to put in entirely unreasonable amounts of time and effort to make it a great conference. What I have to do is provide a structure in which they can be effective, and in which they can feel like they're being effective. I'm very grateful to all the people in TPM who have helped out so much so far, and I know that this help will continue (and accelerate, even!) going from here all the way to the conference, which should be most excellent as a result of all this energy and planning. (The speakers must be commended, too; they're all volunteers who help out YAPC as well, even if they do it in a different way.)

The commercial software project aspect of things is that YAPC has a deadline, and it has a budget, and people are charged money to attend (not a lot, but still it makes for a different dynamic). There is no "release early, release often" aspect of YAPC. There is no "we'll only release it when it's ready" aspect. These constraints drive the process.

I've heard rumors that you have some new events planned. What's this about a scavenger hunt? How about the hackathon beforehand?

RD: You are well-connected to the rumor mill, I see.

The scavenger hunt actually originated with Kevin Meltzer. He and I were chatting on AIM one day and he suggested it to me, seemingly out of the blue. I guess what happened was he got interested in seeing what there was to do in Toronto for personal touristing reasons (people often come a few days early or stay a few days late to YAPCs in order just to see the sights). What he found was that there was just plain too much to do in Toronto, and he got excited about experiencing the city as a whole. This is actually something that the TPM group wanted to accomplish in hosting a YAPC. Toronto is one of the great urban environments of North America (on the scale of San Francisco or Chicago) and YAPC is being held right in the heart of downtown. We're happy and proud for an opportunity to share the city with all the people who are coming in from afar to visit, especially people who haven't been before.

So Kevin made the suggestion, and I thought it was a great idea. At the next TPM meeting I mentioned it to the whole group and everyone seemed really pleased with it. Basically, we'll divide up the conference attendees into groups of 10 and send people out into the city with their digicams to track down and photograph local landmarks and places of geek-interest. This will happen on the night of Monday 27 June. The winners will be announced...

... on Tuesday night, which will be the night of the social centerpiece of YAPC. For those who don't know their geography (or don't happen to have a map handy), Toronto is situated on the north shore of Lake Ontario. Lake Ontario is one of the Great Lakes, which means it's big. Really big. More of an inland sea than a lake. Tuesday night, we're taking a cruise. We've booked a major cruise ship to hold the entire conference, and for 4 hours it will navigate Toronto Harbor, the channels through the Toronto Islands, and the near coast of Lake Ontario. I've done this kind of cruise once before; it makes for a great combination of natural and urban beauty. (The Toronto skyline is one of the most impressive in the world, and it looks great from the lake as day turns to night.) We're also going to be having our banquet dinner on the cruise boat, and our fundraising auction. (There will be some special items to be had during the auction as well. They are sure to raise eyebrows.)

There will be happenings before the conference officially begins, too. Uri Guttman (of is the official YAPC 'socialist czar', and he's arranging for a group restaurant dinner for anyone who arrives to the conference on Sunday (which should be almost everyone from out of town) and he's also trying to book a movie theatre in the downtown area for our exclusive use to see something, we're not quite sure what yet. (What are the big sci-fi films to be released towards the end of June?) We're still looking for sponsorship for this.

Though neither social nor officially part of the conference, the 'lambdacamels' will descend upon Toronto, too. This is the group led by Autrijus Tang, founder of the Pugs project. Pugs is an effort to implement Perl 6 in Haskell as a kind of proof-of-concept for the real Perl 6 (which will run on Parrot), as well as to give Larry a working Perl 6 example through which he can focus on the details in the Perl 6 language design. About a dozen Pugs hackers will show up in Toronto nearly a week before the conference happens, and then be whisked away to a wilderness hideaway where they will do nothing but hack on Pugs for several days before the conference starts. (We have a lot of wilderness in Canada, even near Toronto, and one of the Toronto Mongers has a large cottage where he'll house the whole gang.)

From time to time we will throw them scraps of food. If we are pleased with the results, we'll drive them back to civilization in time for the conference.

How did you go about wrangling sponsors?

RD: I'd say it was a three-edged sword.

Yours, mine, and the truth?

RD: You do not understand, but you will.

All 3 edges have something to do with simply being networked into various IT communities. The least of the 3 was me needing sponsors and flipping through my Rolodex. (Well, Palm Pilot, but same idea.) The first--and crucial--sponsor came as a result of my being active in CLUE, the Canadian Linux Users Exchange. It is an organization that attempts to act as a clearing-house for Linux community activities around Canada. The head of that organization, Matt Rice, is a big Perl programmer and advocate as well. And he is also closely connected with the people at LPI, the Linux Professional Institute.

So one day Matt and I were talking, and I told him how I was mustering TPM to make a YAPC bid. This was in July 2004, making it the 2nd bid that TPM would make for YAPC, as we lost the 2003 bid to Buffalo. Buffalo did a great job, and I was happy to have lost to them after I went to YAPC::NA 2004 there and saw the quality of their event. One of the really big things that they had going for them was that the CompSci department at SUNY Buffalo donated the entire venue to them, gratis. They just had a new building built on campus and they were eager for an opportunity to showcase it. This, plus a desire on the part of the chairman of the CS department there to 'give back' to the open source community in general (and the Perl community in specific) led to the donation of their new building as a venue. It's a really nice building. We were very fortunate to have had it for the conference.

Did this happen before TPF accepted the proposal or after?

RD: It was part of the 2004 bid that submitted to TPF in August 2003. As a member of the Perl community, I was really happy with the donation from the CS department there and how it made for a great YAPC. As a losing bidder for YAPC::NA 2004, though, I felt differently. (Like I said, I'm glad that they did [win]. You can't look that kind of gift horse in the mouth.)

I felt like I had a trump card played against me. We put a lot into our 2004 bid, and yet there was no argument I could make that would have changed the outcome, or should have changed the outcome.

So I felt vulnerable when it came to creating a bid in the summer of 2004 for 2005. Like, no matter how hard I worked, someone else could produce the same magical trump and we would be shut out. So I made finding that same trump card myself to put into the Toronto bid my first order of business. I explained it this way to Matt, and he brought it to some people at LPI, and they realized the value of supporting a great open source community like the Perl community, and they essentially committed to a sponsorship large enough that our venue requirements would be covered.

So LPI became the foundation sponsor of YAPC::NA 2005 in Toronto. I honestly can't imagine how I would have made the conference work without them.

How much of the previous bid did you re-use?

RD: Much of it. We cleaned it up a bit, and added "options" to it. In the previous year I identified one potential venue to put into the bid. For the 2005 bid, I kept that one but added another, in case TPF found one or the other preferable. But the main addition to the bid was the LPI sponsorship. To compare the work effort of the two bids, I'd say that the bid we put together in the summer of 2003 took about a dozen of us about 2.5 weeks to put together, in our off-hours, using a Kwiki. (That was when I first appreciated just how useful wikis can be in collaborative documentation development tasks!)

The bid in 2004, I assembled myself in about 2 days. (Not including the effort of finding the LPI sponsorship or in 'scouting out' the venue possibilities.)

Having potential venues and a real sponsor must have helped.

RD: I don't know if the 2005 bid benefited from having two venue possibilities as opposed to one, but it certainly benefited from having the foundational sponsorship of LPI.

How about the rest of the sponsors?

RD: So I've covered two edges of the sword. The third edge is by random chance. Although it must be said that chance favors the prepared mind.

In September 2004 there was a PHP conference happening in Toronto, and I kept up a degree of correspondence with the producer of that conference, Marco Tabini. (This was as part of the work I was doing for CLUE, which was making myself aware of open source events happening in Canada.) The day that conference was to wrap up, I eked some time out of work and went over to the conference venue to congratulate Marco on his conference. So we finally met in person there, and he invited me to stay and join him for lunch. I did... and promptly got separated from him and the rest of his core group in the buffet line.

So after I got my food, I went to whatever other random table appeared to have a free seat and I got into conversation with the rest of the table. At one point during conversation at the table one of the guys there asked us all what we thought of DB2 (the IBM database product). About a year earlier I had worked with it in a Perl project and I had had a hell of a time, so I heaped a bit of good-natured scorn upon it, mostly having to do with how difficult it was to integrate with Perl. After a few minutes of ranting I asked him why he asked. The reason was because he was the Product Manager for DB2 and Open Source Software at IBM. ($name = 'Dan Scott')

Moral: ask before ranting, unless you're the kind of person who asks and then doesn't rant.

RD: Fortunately, Dan is one of those people who takes criticism constructively! :-) He was really happy to hear that I had an opinion on Perl and DB2, and he wanted to get together with me to get down my thoughts in a more organized fashion, so that he could direct some resources within IBM toward addressing those concerns. So we made an appointment to do that. And then I mentioned to him that there was this YAPC thing that had just recently been awarded to Toronto, and maybe IBM would like to help out with that too. Actually, this may have been a few days before the official announcement that had been awarded YAPC, but Kevin Meltzer and I had talked informally about it before that.

So anyhow, Dan was receptive to the idea, and he put me in touch with the right people at IBM for this kind of thing. It took a few months to move its way through the process there, but with the help of some great sympathetic people at IBM they came through with a tremendous sponsorship, which is how we are able to have the cruise/banquet I told you about earlier. To summarize: direct networking, Rolodex, and random. (The latter two being manifestations of indirect networking, I suppose).

This covers my sponsorship efforts. Since then I have deputized a few people at TPM to try to find some more. That is in process but they've had some promising responses so far.

Is the goal of a YAPC to be self-sufficient or to make a bit of money?

RD: The goal is to be self-sufficient. As I understand it, there is no official TPF position that YAPC should create a surplus that can go back to TPF to help it fund its other goals.

That said, the previous few YAPCs have had surpluses, though not huge ones. I really don't know if YAPC::NA 2005 in Toronto will or won't this year.

Thus every new sponsor you find can fund some other neat thing, like a movie trip or the cruise?

RD: Yes, that's exactly correct. As I mentioned earlier, we're still hoping to find a sponsor for the movie night that Uri is organizing. Sometimes sponsorship money won't be used for "fun" or "conference enhancing" items. For instance, I really don't know whether or not I can provide my conference volunteers with free admission to the conference. I need to find maybe $2000 in order just to cover their variable food costs.

A conference without volunteers definitely counts as "not fun" in my book.

RD: Oh, we'll have volunteers. It's just that, as things stand right now, they'll pay their conference fee just like everyone else. I've paid my conference fee. Many others on the volunteer side of things have already, too. It's like that bit in "Starship Troopers": In Raczak's Roughnecks, everyone jumps, everyone fights!

What were you looking for when you and your team put together the schedule this year?

RD: We kind of didn't know what we were looking for, at first. The call-for-papers (CFP) went active on January 24, I think. Up until maybe three days before the close of CFP (April 18), we had paper submissions totaling maybe only 75% of what was needed to fill the conference, even if we stretched some of the talks out. When the CFP ended, we were up to maybe 250% of what was needed to fill the conference. Yes, there were a LOT of last minute submissions.

I can't manage to feel any surprise!

RD: No, I don't blame you.

But there's a huge difference between being an impassionate outside observer, and being the poor schmuck who won't know whether or not their life has meaning until the CFP totals are in in another couple of days. I was pretty tense leading up to that.

It was pretty obvious to us that there would be certain "tracks" that would form in the conference: Perl 6 and testing were the two big ones. The process of putting together the schedule went something like this:

  • Pick out the very few "absolute-must-have" talks from the stack.
  • Find a time for them in the utterly empty schedule.
  • Now, figure out the talks which thematically support those talks.
  • If there are too many for those themes, then figure out if any of those overlap, and pick one of the overlapping.
  • And now cut the list down by another talk or two, somehow.

So now we have our anchors, and our tracks. That left us with only so much schedule left to fill, and way too many talks to do it with. So we (the program selection committee) went into a kind of frenzied scrum for about two hours with back-and-forth advocating of personal favorites. When the dust cleared, we had our schedule.

I should say that all of this happened on Saturday, April 23, when six of us got together and spent the whole day working through the schedule.

Did you notice other trends>

RD: Other trends... hmm... well, we have a database track, more or less, which is heavily influenced by Class::DBI. Apart from that, I don't know if there is anything we can call a trend, but I can point out a few other features of the schedule.

The opening 1.5 hours of the first day is a plenary session--a bit of time for to welcome everyone who came (and we've got people coming from all over the world!), followed by Larry giving a keynote, followed by Allison giving her "state of the carrot" update on the world of Perl over the past year. The afternoon of day three will be much as it has been the past few years; the Lightning Talks, followed by a closing keynote, followed by a Town Hall meeting.

One interesting thing that happened with the schedule that I was expecting is that, in day two, we are actually going to bump up the number of parallel tracks from three to four, which I think is a first for a YAPC::NA. This is to accommodate a "donated tutorial" from Stonehenge that brian d foy, Stonehenge trainer and originator of the Perl Monger movement, will present. We haven't yet identified the room at the conference venue that we will book for him, so we've been calling it the "brian d foyer". We're all just way too pleased with ourselves for that. Even once we have figured out a room, no doubt that name will stick.

We thought it was important to put this tutorial into the schedule so that newcomers to Perl would have something meaningful at the conference to cater to their needs. This was raised at the Town Hall meeting in Buffalo at the end of YAPC::NA 2004 and so we're trying to take those comments to heart. It has meant finding more money to book the extra room, but we all thought it was important enough to make the stretch to do this.

Do you expect a lot of newcomers? I didn't really catch the breakdown between experienced programmers and neophytes at the last couple of YAPCs.

RD: I wish I was in a better position to answer that. Almost by definition it's hard to predict, as these people don't circulate in the same communication channels that I do. I talked with brian d foy about what his expectations were in terms of the number of people who would attend his introductory Perl tutorial at YAPC, and he said that when he has given similar talks at similar conferences he's always had a good turn out.

That's as close I can get to knowing, I'm afraid. As for previous YAPCs, well, what I recall from Buffalo was that the newcomers were certainly vocal in the Town Hall meeting. That's a good thing! I absolutely hate the idea that the Perl community could degenerate into an echo chamber. New perspectives are necessary.

How is registration going?

RD: Just a second...I'll have a look....

Wow! This was the first time I looked today--seven new since yesterday. Which is huge. It could be the biggest single day jump yet. The total is 106. Which is considerably ahead of any previous YAPC::NAs I have any data for this number of days before the conference starts. (T-minus-YAPC, as it were.)

My data set goes back to YAPC::NA 2002 in St. Louis. 1999 - 2001, I don't know. Maybe the information is out there somewhere within TPF, but I haven't figured out where yet. Right now, my projection for registrants + speakers + volunteers is between 300 and 400.

Do you have access to a lot of that through previous organizers?

RD: Only going back to 2002. I think that's when TPF instituted a new back-end data system.

How many speakers and volunteers do you estimate will be there?

RD: Speakers, I don't have to estimate--the number is 45. (Not including Lightning Talks.) Volunteers--maybe 20? 25? We're in low teens now, and there's a lot of buzz within the larger TPM group so I expect a lot of people will offer their time for the actual conference duration.

Regarding the range, my actual projections are for 340 right now. 300 is being conservative, assuming we lose quite a lot of momentum. My gut tells me that we're actually going to gain a fair bit more. 400 would be a blow-out success.

How many can the venue hold?

RD: The venue can hold 420 before I start getting the facility manager to start lining chairs up against the walls. More importantly, the cruise boat can hold 500. :-) While this is all going by gut, I think we'll be around the 400 total mark. Maybe I'm being influenced by the seven registrants today. Earlier this weekend there was a day when there wasn't a single registrant. That doesn't happen often. Maybe had you asked me on that day I wouldn't have been so optimistic.

How about the hotel and the dorms?

RD:Ahh....there's the rub.

Right now, I have 250 rooms reserved in the same hotel facility that is hosting the conference. I have 100 rooms reserved in a university dorm about 1km away from the conference, for students or other people who are particularly cost-conscious. I also have 50 rooms reserved in a hotel that is part of the same university that has the dorm rooms, so that's also about 1km away from the conference facility. So, we have plenty for the whole conference.


The group reservation for the 250 rooms at the hotel facility expires on May 12. At that point, any unbooked rooms go back on the open market. They're still bookable with the conference code, but they just aren't reserved for us. So they'll probably be sold to the public with a half-life of 4 days.

Toronto is notoriously under-served by hotels, and compounding this is that AA is having its 75th anniversary conference in Toronto on June 30th and July 1. That conference is bringing 79,000 people into the city. The 'thin tail' of AA conference attendees who want to come to Toronto a bit early to do some touristing before their conference starts will put a great deal of strain on the hotel availability in the city.

We call that the long snout. Or the short snout, I forget.

RD: Right! That's a better term. I was wondering what to call it.

Looking at things from the opposite direction, YAPC attendees who decide at the last minute that they want to spend a few extra days in Toronto after the conference ends will be S.O.L. trying to find a hotel. So I'm concerned about all this.

Looking at the registration figures from the previous few years of YAPCs, about 40% of attendees register in the last 2.5 weeks. So for YAPC this year, that would mean 40% registering June 6 and after. There will still be a few rooms left in the dorms and the hotel attached to the university that contains the dorms (our booking arrangements are slightly different with them), but still, there will be a distinct shortage of hotel rooms.

Anything else to say in closing?

RD: YAPC is the Perl conference by Perl programmers, for Perl programmers (and people who think they might like to be Perl programmers). Everyone in the organizing team is doing it because we all feel so fortunate for all the support we have received from our fellow Perl Mongers around the world, and we feel privileged to return the generosity as best we can through the conference.

We hope you can join us in Toronto for the conference, and we're hoping to make it a memorable one for everyone.

A Plan for Pugs

Autrijus Tang is a talented Perl hacker, a dedicated CPAN contributor, and a truly smart man. His announcement of starting an implementation of Perl 6 in Haskell on February 1, 2005 might have seemed like a joke from almost anyone else. A month later, his little experiment runs more code and has attracted a community larger than anyone could have predicted. recently caught up with Autrijus on #Perl6 to discuss his new project: Pugs.

chromatic: I've followed your journal from the beginning, but it didn't start from the start. Where did you come up with this crazy idea?

Autrijus: Ok. The story is that I hacked SVK for many months with clkao. SVK worked, except it is not very flexible. There is a VCS named darcs, which is much more flexible, but is specced using quantum physics language and written in a scary language called Haskell. So, I spent one month doing nothing but learning Haskell, so I could understand darcs. Which worked well; I convinced a crazy client (who paid me to develop Parse::AFP) that Perl 5 is doomed because it has no COW (which, surprisingly, it now has), and to fund me to develop an alternate library using Haskell.

(I mean "Perl 5 is doomed for that task", not "Perl 5 is doomed in general".)

chromatic: Copy-on-Write?

Autrijus: Yeah.

chromatic: So that's a "sort-of has".

Autrijus: Yeah. As in, sky suddenly worked on it and claims it mostly works. Haven't checked the code, though.

chromatic: It's been in the works for years. Or "doesn't works" perhaps.

Autrijus: But I digress. Using Haskell to develop OpenAFP.hs led to programs that eat constant 2MB memory, scale linearly, and are generally 2OOM faster than my Perl library.

Oh, and the code size is 1/10.

chromatic: Okay, so you picked up Haskell to look at darcs to borrow ideas from for svk, then you convinced a client to pay you to write in Haskell and you started to like it. What type of program was this? It sounds like it had a bit of parsing.

Autrijus: AFP is IBM's PDF-like format, born 7 years before PDF. It's unlike PDF in that it's all binary, very bitpacked, and is generally intolerant of errors. There was no free library that parses or munges AFP.

chromatic: Darcs really impressed you then.

Autrijus: The algorithm did. The day-to-day slowness and fragility for anything beyond mid-sized projects did not. But darcs is improving. But yeah, I was impressed by the conciseness.

chromatic: Is that the implementation of darcs you consider slow or the use of Haskell?

Autrijus: The implementation. It basically caches no info and recalculates all unnecessary information. Can't be fast that way.

chromatic: Hm, it seems like memoization is something you can add to a functional program for free, almost.

Autrijus: Yeah, and there are people working on that.

chromatic: But not you, which is good for us Perl people.

Autrijus: Not me. Sorry.

Anyway. So, I ordered a bunch of books online including TaPL and ATTaPL so I could learn more about mysterious things like Category Theory and Type Inference and Curry-Howard Correspondence.

chromatic: How far did you get?

Autrijus: I think I have a pretty solid idea of the basics now, thanks to my math-minded brother Bestian, but TaPL is a very information-rich book.

chromatic: Me, I'm happy just to recognize Haskell Curry's name.

Autrijus: I read the first two chapters at a relaxed pace. By the end of second chapter it starts to implement languages for real and usually by that time, the profs using TaPL as textbook will tell the students to pick a toy language to implement.

chromatic: I haven't seen you pop up much in Perl 6 land though. You seemed amazingly productive in the Perl 5 world. Where'd Perl 6 come in?

Autrijus: As an exercise. I started using Perl 6 as the exercise. I think that answers the first question.

Oh. p6 land.

chromatic: More of a playground than a full land, but we have a big pit full of colorful plastic balls.

Autrijus: Yeah, I was not in p6l, p6i or p6c. However, the weekly summary really helped. Well, because I keep hitting the limit of p5.

chromatic: It seems like an odd fit, putting a language with a good static type system to use with a language with a loose, mostly-optional type system though.

Autrijus: Most of more useful modules under my name, (including the ones Ingy and I inherited from Damian) were forced to be done in klugy ways because the p5 runtime is a mess.

chromatic: You should see Attributes::Scary. Total sympathy here.

Autrijus: Template::Extract uses (?{}) as a nondet engine; PAR comes with its own perlmain.c; let me not mention source filtering. All these techniques are unmaintainable unless with large dosage of caffeine.

chromatic: Yeah, I fixed some of the startup warnings in B::Generate a couple of weeks ago...

Autrijus: Cool. Yeah, B::Generate is abstracted klugery and may pave a way for Pugs to produce Perl 5 code.

chromatic: Parrot has the chance to make some of these things a lot nicer. I'm looking forward to that. Yet you took off down another road.

Autrijus: Actually, I think Pugs and Parrot will meet in the middle. Where Pugs AST meets Parrot AST and the compiler is written in Perl 6 that can then be run on Parrot.

chromatic: I thought Pugs would get rewritten in C for Parrot?

Autrijus: No, in Perl 6.

chromatic: Can GHC retarget a different AST then?

Autrijus: It can, but that's not the easier plan.

chromatic: It's easy for me. I don't plan to do it.

Autrijus: The easier plan is simply for Pugs to have a Compile.hs that emits Parrot AST. Which, I'm happy to discover yesterday, is painless to write. (Ingy and I did a KwidAST->HtmlAST compiler in an hour, together with parser and AST.)

chromatic: Kwid and HTML, the markup languages?

Autrijus: Yeah.

Ok. So back to p6. P5's limit is apparent and not easily fixable

chromatic: It sounds like you wanted something more, and soon.

Autrijus: Parrot is fine except every time I build it, it fails.

chromatic: Try running Linux PPC sometime.

Autrijus: Freebsd may not be a good platform for Parrot, I gathered. Or my CVS luck is really bad. But I'm talking about several months ago.

chromatic: 4.x or 5.x?

Autrijus: 5.x.

chromatic: Ahh, perhaps it was ICU.

Autrijus: Two out of three times is. I think.

chromatic: I guess it's too late to interest you in a Ponie then.

Autrijus: I was very interested in Ponie. I volunteered to Sky about doing svn and src org and stuff, but svn was not kind for Ponie.

obra:Well, that was before svn 1.0

Autrijus: Right. Now it all works just fine, except libsvn_wc, but we have svk now, and I learned that Sky has been addicted to svk.

But anyway. And the beginning stage of Ponie is XS hackery which is by far not my forte. I've read Lathos' book, so I can do XS hackery when forced to but not on a volunteer basis. Oh no.

chromatic: That's a special kind of pain. It's like doing magic tricks, blindfolded, when you have to say, "Watch me push and pop a rabbit out of this stack. By the way, don't make a reference to him yet...."

Autrijus: So, on February 1, when I had too much caffeine and couldn't sleep, I didn't imagine that Pugs would be anything near a complete implementation of Perl 6. I was just interested in modeling junctions but things quickly went out of control. And some other nifty things like subroutine signatures.

chromatic: There's a fuzzy connection in the back of my head about Haskell's inferencing and pattern matching being somewhat similar.

Autrijus: Sure. Haskell has very robust inferencing, pattern matching, and sexy types. Which I'm trying to inflict on luqui to improve Perl 6's design.

chromatic: As long as they do the right thing with regard to roles, go ahead.

Autrijus: They do. :)

chromatic: This was an academic exercise though?

Autrijus: Yeah. It stayed as an academic exercise I think for two days.

chromatic: "Hey, this Perl 6 idea is interesting. I wonder how it works in practice? I bet I could do it in Haskell!"

Autrijus: Yup. Using it as nothing more than a toy language to experiment with, iitially targeting a reduced set of Perl 6 that is purely functional. But by day three, I found that doing this is much easier than I thought.

chromatic: Did you say "highly reduced"?

Autrijus: Yeah. Term is "featherweight".

chromatic: What makes it easier?

Autrijus: Parsec and ContT. Parsec is like Perl 6 rules.

chromatic: Parsec's the most popular Haskell parsing library, right?

Autrijus: Well, Parsec and Happy. Happy is more traditional; you write in a yacc-like grammar thing and it generates a parser in Haskell for you. Parsec is pure Haskell. You just write Haskell code that defines a parser. The term is "parser combinator".

chromatic: Haskell is its own mini-language there.

Autrijus: It's a popular approach, yes. When you see "blah combinator library", think "blah mini-language".

chromatic: I looked at the parser. It's surprisingly short.

Autrijus: And yet quite complete. Very maintainable, too.

chromatic: Now I've also read the Perl 5 parser, in the sense that I picked out language constructs that I recognized by name. Is it a combination parser/lexer, or how does that work? That's the tricky bit of Perl 5, in that lexing depends on the tokens seen and lots of context.

Autrijus: Yup. It does lexing and parsing in one pass, with infinite lookahead and backtracking. Each lexeme can define a new parser that works on the next lexeme.

chromatic: Does that limit what it can do? Is that why it's purely functional Perl 6 so far?

Autrijus: The purely functional Perl 6 plan stops at day 3. We are now fully IO. Started with say(), and mutable variables, and return(), and &?CALLER_CONTINUATION. So there's nothing functional about the Perl 6 that Pugs targets now :).

chromatic: Does Haskell support continuations and all of those funky things?

Autrijus: Yes. And you can pick and match the funky things you want for a scope of your code. "In this lexical scope I want continuations"; dynamic scope, really. "In that scope I want a logger." "In that scope I want a pad."

chromatic: Performance penalty?

Autrijus: Each comes with its own penalty, but is generally small. GHC, again, compiles to very fast C code.

chromatic: Can you instrument scopes at runtime too?

Autrijus: Sure. &?CALLER::SUB works. And $OUTER::var.

chromatic: Are you compiling it to native code now? I remember that being a suggestion a few days ago.

Autrijus: Pugs itself is compiled to native code; it is still evaluating Perl 6 AST, though.

chromatic: It's like Perl 5 in that sense then.

Autrijus: Yes, it's exactly like Perl 5. Have you read PA01?

chromatic: I have.

Autrijus: Cool. So yeah, it's like Perl 5 now. The difference is B::* is trivial to write in Pugs

chromatic: Except maintainable.

Autrijus: And yeah, there's the maintainable bit. Pugs is <4k lines of code. I think porting Pugs to Perl 6 will take about the same number of lines, too.

chromatic: You already have one module, too.

Autrijus: Yup. And it's your favorite module.

chromatic: I've started a few attempts to write Test::Builder in Parrot, but I'm missing a few pieces. How far along are classes and objects in Pugs?

Autrijus: They don't exist. 6.2.x will do that, though. But the short term task is to get all the todo_() cleaned. which will give us an interpreter that really agrees with all synopses. At least in the places we have implementation of, that is.

chromatic: I see in the dailies that you are producing boatloads of runnable Perl 6 tests.

Autrijus: Yup, thanks to #Perl6. I seldom write tests now :) The helpful committers do that for me.

chromatic: How do you know your code works then?

Autrijus: I just look at newest todo_ and start working on it.

chromatic: Oh, they write tests for those before you implement them?

Autrijus: Yup. It's all test-first.

chromatic: Okay, I'll let you continue then.

Autrijus: Ha. So yeah, the cooperation has been wonderful. Camelfolks write tests and libraries, and lambdafolks makes those tests pass. If a camelfolk wants a particular test to pass sooner, then that person can learn from lambdafolk :). Things are easy to fix, and because of the coverage there's little chance of breaking things. If lambdafolks want to implement new things that may or may not agree with synopses or p5 norm, then they learn from camelfolks.

chromatic: Have you started giving Haskell tutorials? I know Larry and Patrick have started to pick up some of it. I'm pretty sure Luke and Damian have already explored it (or something from the same family tree).

Autrijus: I think I've read a paper from Damian that says he taught Haskell in monash. It's before the monadic revolution though.

chromatic: If not Haskell, certainly something from the ML family.

Autrijus: Right. So, I've been pointing people to YAHT and #Haskell.

chromatic: It sounds like you're attracting people from both sides of the fence then.

Autrijus: It indeed is. I get svn/svk patches and darcs patches.

chromatic: Is there a lot of overlapping interest? Where does it come from?

Autrijus: Well, ever since the monadic revolution of '98 Haskell people have started to do real world apps.

chromatic: Now that they can do IO, for example.

Autrijus: Yeah. It's been only 7 years ago. And recently Haskell world has its native version control system; a Perl-review like magazine, cpan/makemaker-like infrastructure, etc. So it's growing fast.

chromatic: There's still a lot of attraction there for real world applications, of which Pugs is one?

Autrijus: Pugs is a practical project in that working on it has a chance of solving real problems, and is very fun to boot. And although p5 got no respect, in general p6 is very slick. So the mental barrier is lower for lambdafolks to join, I think.

chromatic: The lambdafolks like what they see in Perl 6?

Autrijus: Yup. I quoted Abigail on #Haskell a while ago.

chromatic: I saw something earlier about access to libraries and such. Do you have a plan for the XS-alternative?

Autrijus: Yeah, Ingy is working on it ext/Kwid/ eventually inline Haskell code. And with luck, inline other kinds of code as well through Haskelldirect (the Haskell equiv of Inline).

chromatic: Is this within Pugs or Perl 6 atop Pugs?

Autrijus: It's within Pugs. The Parrot side had not been well-discussed.

chromatic: Yeah, the Parrot AST needs more documentation.

You're devoting a lot of time to it. Obra mentioned that you've cleared most of your paying projects out of the way for the time being. What's the eventual end?

Autrijus: And whither then? I cannot say :). As you mentioned, I've diverted most of my paying projects away so I should have at least 6 months for Pugs.

chromatic: How about in the next month?

Autrijus: This month should see robust semantics for basic operations, the beginning of classes and objects, and many real modules hooks to Haskell-side libraries.

chromatic: I'll do T::B then.

Autrijus: Oh and Pugs hands out committer bit liberally so if you want to do T::B, I'll make you a committer :). You can start now actually. Just write imaginary Perl 6 code, and we'll figure out how to make it run. Most of the examples/* started that way.

chromatic: Ah, I'll take a look.

Autrijus: Oh. Right. I was quoting Abigail.

"Programming in Perl 5 is like exploring a large medieval castle, surrounded by a dark, mysterious forest, with something new and unexpected around each corner. There are dragons to be conquered, maidens to be rescued, and holy grails to be quested for. Lots of fun."

"Perl 6 looks like a Louis-XVI castle and garden to me. Straight, symmetric, and bright. There are wigs to be powdered, minuets to be danced, all quite boring.".

I, for one, am happy for Perl to move from the dark age to the age of enlightenment. I think many camelfolks and lambdafolks share the same sentiment :).

The Phalanx Project

Imagine a city protected by a small army of soldiers. The city's future growth requires a larger force; so a few determined lieutenants go to nearby towns and enlist aid from their police departments. These forces will come to the aid of the larger city when the time comes.

This is the Phalanx project, but the city is Perl, our soldiers are automated tests, and the nearby towns are the modules of CPAN.

Flashback to OSCON 2003. Larry Wall had just given his 7th annual State of the Onion where he'd announced the Ponie project. Ponie is to be Perl 5.10, but running on the new Parrot virtual machine that forms the basis of Perl 6, instead of C.

I was talking with Leon Brocard about the massive amount of testing that would be necessary to test a new implementation of Perl. Everything we know, and all our assumptions, would change. How would we know that 2+2=4 all the time? How would we know that object inheritance works? Will XS modules work the way they should? We would need a huge test suite, more than Perl has now, to make sure Ponie really is still Perl 5. The CPAN would make a great source of real-world testing.

Most CPAN modules of any popularity come with a test suite, so it would be easy to add more tests to the distributions. This would help those who worked on Ponie to make sure they had more and more tests to test against, and would help the module author by having more tests written for his code.

Which Modules?

I didn't imagine that we'd run Ponie against all of the CPAN, and wanted to follow the Pareto principle and go after the top 10%. However, with CPAN at about 4,000 modules when Phalanx started (now 6,000), it would have been too large an effort to work on 400 modules. Instead, I picked a nice round 100, or 2.5% of the distributions available.

What makes a "top 100 module"? Ideally, I'd like to know which modules had the most real-life use, but that's impossible. I decided a relative comparison of the number of downloads of modules would be a close enough approximation. (The astute reader is probably already thinking of problems with this approach, but rest assured that I've thought of them as well.)

The Phalanx Top 100 Modules are those most downloaded in 45 days from the main CPAN mirror, with some adjustments. I excluded search engine bots and anything that was apparently mirroring CPAN. I also made the executive decision that any given IP address that downloaded more then 450 modules in 45 days was a bot.

Why the Name "Phalanx"?

In ancient Greece, the phalanx was a military formation where hundreds of soldiers formed a shield wall. Each man stood shoulder to shoulder with the men next to him, their shields overlapping. As it is with the shields of the men in the phalanx, it is with the numerous and overlapping tests of the Phalanx project.

For any set of code, the more automated tests you have, the more protection you have. If you can write a test for something, you probably should. Consider these simple tests of a Project object's constructor and an accessor, tested with Perl's testing framework:

my $project = Project->new( name => "Phalanx" );
isa_ok( $project, "Project" );
is( $project->name, "Phalanx", "Name set correctly" );

Some might say, "It's only an accessor, why should we test it?" It's worth testing because when it doesn't work in production, you won't see the error at the point of the accessor. Instead, some piece of code that uses the Project::name accessor will fail, and you'll spend hours tracing the failure back to the accessor.

This sort of approach -- strength in numbers, each test building on others -- was the basis of the phalanx. So, too, will it be with Perl's tests.


The primary goal of Phalanx is to increase the quality of a given module's test suite, mostly by increasing the amount of the module's code that the tests cover. However, there are secondary goals because we're working with the code anyway.

The first sub-goal is to find hidden bugs. As we add tests to modules, we hope to uncover broken functionality. Indeed, the team working on HTML::TreeBuilder uncovered a bug in the module's code while they added tests.

In addition to adding to the testing, team members should verify the code's documentation and fill in any missing areas. Comparing code to inline documentation may uncover hidden features that only someone reading the code would know about. These should be documented and tested.

The principle here is this: Code, tests, and documentation must all agree with each other. If the code does one thing, the documentation describes it accurately, but the tests check for a different behavior, then there's a bug. It might even be the two that agree with each other that are wrong. It's possible even to find that all three might disagree with each other. Old code can be like that sometimes.

Two other sub-goals are about humans. Phalanx provides an easy way for people to wet their feet in the open source process. The very nature of Phalanx is collaborative, where each team working on a module submits patches to the module for review and approval. The module's author still maintains control, but works with the team to decide what direction testing should take.

Second, Phalanx provides a playground for people with an interest in automated testing who don't know how or where to start. Like chromatic's Perl testing kata, adding tests to existing code actually exercises each team member's skills.

Getting People to Sign Up

Once I'd created the Phalanx 100 and the guiding principles, and put up the Phalanx website, I had to find some hoplites. (Hoplites are the Greek soldiers that made up the ancient phalanxes.) I announced the project and a dozen eager hoplites volunteered. Each hoplite wrote to the author about his intent, to make sure the author was onboard with the idea. No sense in making changes and preparing patches for an author who will reject them. The author may also have input and suggestions, such as areas in the code that need more attention than others. Once the preparation was complete, the hoplite was to add tests and verify documentation.

This process turned out to be a dismal failure.

Twelve different hoplites adopted 12 distributions and produced exactly zero code in the first year. I don't mind pointing fingers, because I was one of the 12. It seems that on projects like this, working solo means motivation is hard to maintain. Each of the hoplites I talked to explained that he started with the best of intentions, but had trouble finding the time to follow through, and the motivation fell by the wayside.

This year, I tried a different approach, enlisting the support of Perl Mongers groups, starting with my home group, I then took to the conference circuit, giving lightning talks at YAPC::NA and OSCON asking for interested parties to join up with the team. Since then,,, and Perl Seminar New York have all joined up. We still coordinate with the module author, and also report progress centrally at our new Phalanx wiki, but now I hope that with a group, it will be easier to keep motivation high.

Phalanx Tools

Over time, as we've built up an infrastructure for Phalanx, three tools have proven themselves to be crucial to collaboration.

First were the triplets of email, web, and wiki, which allow information to be swapped on progress. The perl-qa mailing list hosted at is home to many Perl folks interested in testing. The Phalanx webpage lets me post information for all hoplites to see. The Phalanx wiki allows hoplites and groups to post project progress.

Second, centralized version control is crucial since we have multiple collaborators on an individual module,. Fortunately, Robert and Ask of are graciously hosting a Subversion repository for the Phalanx teams.

Third, Paul Johnson's excellent Devel::Cover package has been invaluable in identifying shortcomings of test suites. Devel::Cover analyzes the running of the tests, and then identifies which lines of code the suite has exercised or "covered." If a line of code isn't covered by a test, it provides the hoplites a great place to start, by writing a test case to exercise the uncovered code.

Devel::Cover presents metrics on percentages of coverage, but Phalanx doesn't try necessarily to increase coverage to 100%. We've found that there's a level of diminishing returns when exercising extreme corner cases, especially cases based on platform-specific dependencies. What we've found is that the real value is finding the big areas of neglect and patching those up. Sometimes you can even find big wins, like when I found unused and un-removed code in WWW::Mechanize.

How You Can Join

If automated testing interests you, or you're looking for a way to add to the CPAN, we'd love to have you join.

  • Join the perl-qa list.

    The perl-qa list is the official mailing list of the Phalanx project. Sign up and introduce yourself.

  • Find a module that interests you.

    Find a module that could benefit from your attention. Many hoplites pick modules that they use in day-to-day life. There's also no requirement that the module you work on is from the Phalanx 100.

  • Find kindred souls.

    Phalanx seems to go better when hoplites team up to work together.

You can (and should) join our ranks and add to our numbers, as we help take Perl that much closer to Perl 6.

Other Links

Visit the home of the Perl programming language:

Sponsored by

Powered by Movable Type 5.02