One of the most important things you can do in a free software project, besides writing code, is to get your key contributors together as often as possible.

I’ve been fortunate to be able to organise a number of events in the past 10 years, and also to observe others and learn from them over that time. Here are some of the lessons I’ve learned over the years from that experience.

Venue

The starting point for most meetings or conferences is the venue. If you’re getting a small group (under 10 people) together, then it is usually OK just to pick a city, and ask a friend who runs a business or is a college professor to book a room for you. Or use a co-working space. Or hang out in someone’s house, and camp in the garden. Once you get bigger, you may need to go through a more formal process.

If you’re not careful, the venue will be a huge expense, and you’ll have to find that money somewhere. But if you are smart, you can manage a free venue quite easily.

Here are a few strategies you might want to try:

• Piggy-back on another event – the Linux Foundation Collaboration Summit, OSCON, LinuxTag, GUADEC and many other conferences are happy to host workshops or meet-ups for smaller groups. The GIMP Developers Conference in 2004 was the first meet-up that I organised, and to avoid the hassle of dealing with a venue, finding a time that suited everyone, and so on, I asked the GNOME Foundation if they wouldn’t mind setting aside some space for us at GUADEC – and they said yes.Take advantage of the bigger conference’s organisation, and you get the added benefit of attending the bigger conference at the same time!
• Ask local universities for free rooms – This won’t work once you go over a certain size, but especially for universities which have academics who are members of the local LUG, they can talk their department head into booking a lecture theatre & a few classrooms for a weekend. Many universities will ask to do a press release and get credit on the conference web-site, and this is a completely fair deal.The first Libre Graphics Meeting was hosted free in CPE Lyon, and the GNOME Boston Summit has been hosted free for a number of years in MIT.
• If the venue can’t be free, see if you can get someone else to pay for it – Once your conference is bigger than about 200 people, most venues will require payment. Hosting a conference will cost them a lot, and it’s a big part of the business model of universities to host conferences when the students are gone. But just because the university or conference center won’t host you for free doesn’t mean that you have to be the one paying.Local regional governments like to be involved with big events in their region. GUADEC in Stuttgart, the Gran Canaria Desktop Summit, and this year’s Desktop Summit in Berlin have all had the cost of the venue covered by the host region. An additional benefit of partnering with the region is that they will often have links to local industry and press – resources you can use to get publicity and perhaps even sponsorship for your conference.
• Run a bidding process – by encouraging groups wishing to host the conference to put in bids, you are also encouraging them to source a venue and talk to local partners before you decide where to go. You are also putting cities in competition with each other, and like olympic bids, cities don’t like to lose competitions they’re in!

Budget

Conferences cost money. Major costs for a small meet-up might be
covering the travel costs of attendees. For a larger conference, the
major costs will be equipment, staff and venue.

Every time I have been raising the budget for a conference, my rule of
thumb has been simple:

1. Decide how much money you need to put on the event
2. Fundraise until you reach that amount
3. Stop fundraising, and move on to other things.

Raising money is a tricky thing to do. You can literally spend all of
your time doing it. At the end of the day, you have a conference to put
on, and the amount of money in the budget is not the major concern of
your attendees.

Remember, your primary goal is to get project participants together to
advance the project. So getting the word out to prospective attendees,
organising accommodation, venue, talks, food and drinks, social
activities and everything else people expect at an event is more
important than raising money.

Of course, you need money to be able to do all the rest of that stuff,
so finding sponsors, fixing sponsorship levels, and selling your
conference is a necessary evil. But once you have reached the amount of
money you need for the conference, you really do have better things to
do with your time.

There are a few potential sources of funds to put on a conference – I
recommend a mix of all of these as the best way to raise your budget.

• Attendees – While this is a controversial topic among many communities, I think it is completely valid to ask attendees to contribute something to the costs of the conference. Attendees benefit from the facilities, the social events, and gain value from the conference.Some communities consider attendance at their annual event as a kind of reward for services rendered, or an incitement to do good work in the coming year, but I don’t think that’s a healthy way to look at it.There are a few ways for conference attendees to fund the running of the conference:
  1. Registration fees – This is the most common way to get money from conference attendees. Most community conferences ask for a token amount of fees. I’ve seen conferences ask for an entrance fee of €20 to €50, and most people have not had a problem paying this.A pre-paid fee also has an additional benefit of massively reducing no-shows among locals. People place more value on attending an event that costs them €10 than one where they can get in for free, even if the content is the same.
  2. Donations – very successfully employed by FOSDEM. Attendees are offered an array of goodies, provided by sponsors (books, magazine subscriptions, t-shirts) in return for a donation. But those who want can attend for free.
  3. Selling merchandising – Perhaps your community would be happier hosting a free conference, and selling plush toys, t-shirts, hoodies, mugs and other merchandising to make some money. Beware: in my experience you can expect less from profits from merchandising sales than you would get giving a free t-shirt to each attendee with a registration fee.
• Sponsors – Media publications will typically agree to “press sponsorship” – providing free ads for your conference in their print magazine or website. If your conference is a registered non-profit which can accept tax-deductible donations, offer press sponsors the chance to invoice you for the services and then make a separate sponsorship grant to cover the bill. The end result for you is identical, but it will allow the publication to write off the space they donate to you for tax.What you really want, though, are cash sponsorships. As the number of free software projects and conferences has multiplied, the competition for sponsorship dollars has really heated up in recent years. To maximise your chances of making your budget target, there are a few things you can do.
  1. Conference brochure – Think of your conference as a product you’re selling. What does it stand for, how much attention does it get, how important is it to you, to your members, to the industry and beyond? What is the value proposition for the sponsor?You can sell a sponsorship package on three or four different grounds: perhaps conference attendees are a high-value target audience for the sponsor, perhaps (especially for smaller conferences) the attendees aren’t what’s important, it’s the attention that the conference will get in the international press, or perhaps you are pitching to the company that the conference is improving a piece of software that they depend on.Depending on the positioning of the conference, you can then make a list of potential sponsors. You should have a sponsorship brochure that you can send them, which will contain a description of the conference, a sales pitch explaining why it’s interesting for the company to sponsor it, potentially press clippings or quotes from past attendees saying how great the conference is, and finally the amount of money you’re looking for.
  2. Sponsorship levels – These should be fixed based on the amount of money you want to raise. You should figure on your biggest sponsor providing somewhere between 30% and 40% of your total conference budget for a smaller conference. If you’re lucky, and your conference gets a lot of sponsors, that might be as low as 20%. Figure on a third as a ball-park figure. That means if you’ve decided that you need €60,000 then you should set your cornerstone sponsor level at €20,000, and all the other levels in consequence (say, €12,000 for the second level and €6,000 for third level).For smaller conferences and meet-ups, the fundraising process might be slightly more informal, but you should still think of the entire process as a sales pitch.
  3. Calendar – Most companies have either a yearly or half-yearly budget cycle. If you get your submission into the right person at the right time, then you could potentially have a much easier conversation. The best time to submit proposals for sponsorship of a conference in the Summer is around October or November of the year before, when companies are finalising their annual budget.If you miss this window, all is not lost, but any sponsorship you get will be coming out of discretionary budgets, which tend to get spread quite thin, and are guarded preciously by their owners. Alternatively, you might get a commitment to sponsor your July conference in May, at the end of the first half budget process – which is quite late in the day.
  4. Approaching the right people – I’m not going to teach anyone sales, but my personal secret to dealing with big organisations is to make friends with people inside the organisations, and try to get a feel for where the budget might come from for my event. Your friend will probably not be the person controlling the budget, but getting him or her on board is your opportunity to have an advocate inside the organisation, working to put your proposal in front of the eyes of the person who owns the budget.Big organisations can be a hard nut to crack, but free software projects often have friends in high places. If you have seen the CTO or CEO of a Fortune 500 company talk about your project in a news article, don’t hesitate to drop him a line mentioning that, and when the time comes to fund that conference, a personal note asking who the best person to talk to will work wonders. Remember, your goal is not to sell to your personal contact, it is to turn her into an advocate to your cause inside the organisation, and create the opportunity to sell the conference to the budget owner later.

  Also, remember when you’re selling sponsorship packages that everything which costs you money could potentially be part of a sponsorship package. Some companies will offer lanyards for attendees, or offer to pay for a coffee break, or ice-cream in the afternoon, or a social event. These are potentially valuable sponsorship opportunities and you should be clear in your brochure about everything that’s happening, and spec out a provisional budget for each of these events when you’re drafting your budget.

Content

Conference content is the most important thing about a conference. Different events handle content differently – some events invite a large proportion of their speakers, while others like GUADEC and OSCON invite proposals and choose talks to fill the spots.

The strategy you choose will depend largely on the nature of the event. If it’s an event in its 10th year with an ever increasing number of attendees, then a call for papers is great. If you’re in your first year, and people really don’t know what to make of the event, then setting the tone by inviting a number of speakers will do a great job of helping people know what you’re aiming for.

For Ignite Lyon last year, I invited about 40% of the speakers for the first night (and often had to hassle them to put in a submission, and the remaining 60% came through a submission form. For the first Libre Graphics Meeting, apart from lightning talks, I think that I contacted every speaker except 2 first. Now that the event is in its 6th year, there is a call for proposals process which works quite well.

Schedule

Avoiding putting talks in parallel which will appeal to the same people is hard. Every single conference, you hear from people who wanted to attend talks which were on at the same time on similar topics.

My solution to conference scheduling is very low-tech, but works for me. Coloured post-its, with a different colour for each theme, and an empty talks grid, do the job fine. Write the talk titles one per post-it, add any constraints you have for the speaker, and then fill in the grid.

Taking scheduling off the computer and into real life makes it really easy to see when you have clashes, to swap talks as often as you like, and then to commit it to a web page when you’re happy with it.

I used this technique successfully for GUADEC 2006, and Ross Burton re-used it in 2007.

Parties

Parties are a trade-off. You want everyone to have fun, and hanging out is a huge part of attending a conference. But morning attendance suffers after a party. Pity the poor community member who has to drag himself out of bed after 3 hours sleep to go and talk to 4 people at 9am after the party.

Some conferences have too many parties. It’s great to have the opportunity to get drunk with friends every night. But it’s not great to actually get drunk with friends every night. Remember the goal of the conference: you want to encourage the advancement of your project.

I encourage one biggish party, and one other smallish party, over the course of the week. Outside of that, people will still get together, and have a good time, but it’ll be on their dime, and that will keep everyone reasonable.

With a little imagination, you can come up with events that don’t involved loud music and alcohol. Other types of social event can work just as well, and be even more fun.

At GUADEC we have had a football tournament for the last number of years. During the OpenWengo Summit in 2007, we brought people on a boat ride on the Seine and we went on a classic 19th century merry-go-round afterwards. Getting people eating together is another great way to create closer ties – I have very fond memories of group dinners at a number of conferences. At the annual KDE conference Akademy, there is typically a Big Day Out, where people get together for a picnic, some light outdoors activity, a boat ride, some sightseeing or something similar.

Extra costs

Watch out for those unforeseen costs! One conference I was involved in, where the venue was “100% sponsored” left us with a €20,000 bill for labour and equipment costs. Yes, the venue had been sponsored, but setting up tables and chairs, and equipment rental of whiteboards, overhead projectors and so on, had not. At the end of the day, I estimate that we used about 60% of the equipment we paid for.

Conference venues are hugely expensive for everything they provide. Coffee breaks can cost up to $10 per person for a coffee & a few biscuits, bottled water for speakers costs $5 per bottle, and so on.  Rental of an overhead projector and mics for one room for one day can cost €300 or more, depending on whether the venue insists that equipment be operated by their a/v guy or not.

When you’re dealing with a commercial venue, be clear up-front about what you’re paying for.

On-site details

I like conferences that take care of the little details. As a speaker, I like it when someone contacts me before the conference and says they’ll be presenting me, what would I like them to say? It’s reassuring to know that when I arrive there will be a hands-free mic and someone who can help fit it.

Taking care of all of these details needs a gaggle of volunteers, and it needs someone organising them beforehand and during the event. Spend a lot of time talking to the local staff, especially the audio/visual engineers.

In one conference, the a/v guy would switch manually to a screen-saver at the end of a presentation. We had a comical situation during a lightning talk session where after the first speaker, I switched presentations, and while the next presentation showed up on my laptop, we still had the screensaver on the big screen. No-one had talked to the A/V engineer to explain to him the format of the presentation!

So we ended up with 4 Linux engineers looking at the laptop, checking connections and running various Xrandr incantations, trying to get the overhead projector working again! We eventually changed laptops, and the a/v engineer realised what the session was, and all went well after that – most of the people involved ended up blaming my laptop.

Have fun!

Running a conference, or even a smaller meet-up, is time consuming, and consists of a lot of detail work, much of which will never be noticed by attendees. I haven’t even dealt with things like banners and posters, graphic design, dealing with the press, or any of the other joys that come from organising a conference.

The end result is massively rewarding, though. A study I did last year of the GNOME project showed that there is a massive project-wide boost in productivity just after our annual conference, and many of our community members cite the conference as the high point of their year.

So – how can you make a change in a Linux-based distribution? Here’s what happens when everything works as it should:

1. You open a bug report for the feature against your distribution
2. You identify the module or modules you need to change to implement the new feature
3. You open bug reports for each of the modules concerned, detailing the feature and the changes needed in that module for the feature
4. You write a patch to implement the feature, and propose it (appropriately cut up for ease of review) to the maintainers of those modules
5. Once the code has gone through the appropriate review process, it will be committed to the source control of the module(s)
6. Some time later, the maintainer of each module will include that code in a stable release of the module
7. Some time after that, the new stable versions will be packaged and uploaded to MeeGo
8. Your code will be included in the next release of the distribution following the upload.

When people talk about “working upstream” in MeeGo or Linaro, this is what they mean.

To simplify matters for our analysis, let’s consider that the feature we want to implement is self-contained in one module (or related modules which release together). There are two different scenarios we’ll consider:

1. The module is maintained by people not associated with your distribution (for example, a GNU or GNOME project)
2. The module is maintained by people closely related to your distribution (for example, Unity in Ubuntu, or oFono in MeeGo)

We will also look at a third situation, where you find and fix a bug in the software you are using – that is, a released version of a distribution (the proverbial “scratching an itch”).

For each case, I will try to pick a representative feature/patch and follow it from developer through to distribution to Real Users.

What if your code changes different projects?

If your code touches several modules (for example, if you are proposing some new API in GTK+ which you want to use in the GIMP) then things can get complicated – you will need a stable version of GTK+ to be released before you can ship a stable release of the GIMP which depends on it.

This issue of staggered releases is one that Andrew Cowie pointed out a few years ago for language bindings. To avoid making bindings on shifting sands, he preferred to package new APIs once they had been included in a stable GNOME release. In turn, Java GNOME developers rarely depend on development release bindings, and they would wait for the new API to be included in a stable bindings release. For example, the gtk_orientable_get_orientation, added to GTK+ at the end of September 2008, was released in GTK+ 2.16, in March 2009. The first version of Java-GNOME which depended on GTK+ 2.16 was version 4.0.13, released in August 2009. That was packaged in distributions in Autumn 2009, and so most users would not have access to the newer bindings for a few months after that – perhaps early 2010 – at which point, the API was written 18 months beforehand.

And that is when you have a regular release schedule you can rely on! Pity the developer who wants to release a GIMP plug-in which depends on some API included in GIMP 2.8 – the last stable GIMP release, 2.6, came out in October 2008, and over two years later, 2.8 still has not released. And when you combine unreliable release schedules for distributions and applications, the results are cumulative: users of the stable Debian distribution are still using GIMP 2.4 releases. The GIMP 2.4 released in October 2007. Features added to the GIMP in late 2007 are still not in the hands of users of stable Debian distributions.

Getting features to users

It is difficult to generalise when users upgrade their Linux distributions, or even to say what proportion of Linux users are new users at any given time. It would be over-simplifying to say that developers use bleeding-edge distributions, power users upgrade early to the latest and greatest, new users install the latest distributions available, but will only upgrade every 18 months or so afterwards, and conservative users stick with “Long term service” or stable distributions. Most developers I know use their computer for work (and thus want a stable distribution) and only install the latest versions of various dependencies they need to work on their project. But let’s generalise and say that this is roughly the case. So (guesstimating) about 10% of your users will be upgrading to the latest distribution very quickly after its release, a further 20% in the months after when the bugs are shaken out, and the rest will follow along in their own time, perhaps 12 or 18 months later.

To make this concrete, let’s follow the life of a single patch. This is complete anecdata, but in my defence, the patch has been chosen by random, from a project which I know has good community processes and release management in place. The patch we’re going to follow adds an extension to Inkscape to render objects along triangular paths.

1. Bug #226001 opened on 2008-05-03 by inductiveload, with a description of the feature to be added, and proposed code to implement it. The code, as an extension, may have a lower bar for acceptance than code which is core to a project.
2. Patch submission reviewed on 2008-05-03, minor comments, but patch is accepted (note: This was not the authors first submission to Inkscape)
3. Patch corrected to respond to comments and committed on 2008-05-03 (did I mention these guys had good community processes!?!)
4. Inkscape 0.47-pre0, containing the Triangle extension, released on 2009-07-02
5. Inkscape 0.47-pre4 included in Ubuntu 9.10

So for a feature developed in mid 2008, most Inkscape users will still not have the feature by the end of 2009, 18 months later. This is both a typical and atypical example: in many projects, patch proposals lay unreviewed for days, weeks, sometimes months, but the 0.47 release cycle was a particularly long one for Inkscape. However, I think the lag from code written to presence on user’s hard drives of ~12 to 18 months is about correct.

Does it have to be this hard?

If this were the only way to get features into a distribution, trying to improve MeeGo by contributing upstream would be a very frustrating experience. Happily, there are ways to accelerate the process. Taking the MeeGo kernel as an example, where Greg Kroah-Hartman recently threw in the towel on persuading people to propose patches upstream; the process is supposed to work like this:

1. Propose a patch for inclusion upstream. This patch will then ship in a future stable kernel release (let’s say 2.6.38).
2. After peer review, when the code has been accepted for inclusion in the kernel upstream, propose a backport for inclusion in the MeeGo kernel. The back-ported patch will be maintained across the next MeeGo release, and will be dropped when the kernel version included in the MeeGo project catches up with 2.6.38.

The overhead here is reduced basically to the peer review process of the upstream project, and the cumulative cost of merging a patch over the course of 6 months.

As a distributor (or a developer working on a specific distribution), this allows you to get code to everyone, eventually, and have that code included in your distribution as soon as you are sure that it is up to the standard expected by the community. Currently in MeeGo, the trend seems to be more towards submitting patches concurrently upstream and to MeeGo kernel maintainers (or even submitting them upstream once they have been accepted into the MeeGo kernel). In the case that a patch requires substantial modifications, or is rejected outright, upstream, the kernel maintainers are then left carrying a patch indefinitely in the distribution. For one patch, this might not be a big deal, but for thousands of patches, the maintenance and integration burden of these patches adds up.

It is also not unusual for kernel developers to maintain their own git branches for a long time. Three examples that come to mind are inotify, which Robert Love maintained for over a year for both Novell and in the kernel before it was accepted into the mainline, ReiserFS, which was maintained for several years out-of-tree before being shipped with the Linux kernel in 2001, and the fast  desktop patchset which Con Kolivas maintained for almost five years on the -ck kernel branch. Distributions will occasionally ship a substantial diff to upstream if there is a maintainer committed to getting the code upstream eventually. Allocating someone to work over a long period to make everyone happy and comfortable with your code may enable you to ship a big patch to upstream, but this will not be sustainable long term.

To summarise: when working upstream, as a distribution, you should only ship with patches which have been accepted in a development version of upstream already, if you can help it.

Meetings in telephone boxes

Sometimes, however, when upstream and downstream coincide, you can simplify things considerably, while also adding a small measure of risk.

In MeeGo, to continue with that example, the distribution architects have a pretty good idea when they can expect emergency telephony to be ready for oFono and the MeeGo telephony stack, because they’re writing it. By co-ordinating the upstream release management with downstream packaging, you can make promises as a distyribution which you can’t with community-developed software.

When upstream and downstream are co-ordinating each other, we cut out the middleman. The workflow becomes:

1. Report a bug/feature request against a component of the distribution
2. Develop a patch which implements the feature, and submit it directly to the distribution bug tracker
3. Once it has been reviewed and accepted, you know that your patch will be included in the next version of the distribution.

This gives a distribution much more control, both over what gets done, and when, and explains both the Ayatana and MeeGo UX development projects. However, being able to plan around the release is no guarantee that the release will happen on time: GNOME has in the past been stung by planning during the 2.6 development cycle to depend on a new version of GTK+, only to find that the release was delayed. In the end, the GTK+ release shipped in time for the 2.6 release at the end of March.

Scratch scratch

The other patch lifecycle I’d like to mention, because it is so relevant to distributions, was pointed out to me by Federico Mena Quintero yesterday. What happens to a patch that someone makes and submits to a distribution when they find a bug in stable released software? This is one of the key advantages of free software – if you find a bug in the software you use, and you have the wherewithall, you can fix the bug and share that fix with everyone else.

However, as we have seen, there is typically a lag of several months from the time that software is released and the time it is being used by large numbers of users through distributions. With releases of Red Hat Enterprise Linux, Novell Suse Linux Desktop and Ubuntu LTS being supported for up to 5 years, it is possible that important bugs will be fixed in these stable versions for years after the original developers have moved on, and are no longer maintaining older stable versions.

Let’s say I find and fix a bug in Rhythmbox 0.12.5, which ships with Ubuntu 9.10. I open a bug report on Launchpad, attach a fix to the source .deb there, and I update my local copy. As a user, I’m happy – I have fixed my problem and shared the solution with others. If I’m particularly conscientious, I might open a bug on gnome.org against Rhythmbox and attach my patch there, but since the development version is now 0.13.2, the best you can hope for is that the patch applies cleanly to the master branch, and will be included in the next release. It is very unlikely that the upstream maintainers will release another update to the 0.12 series at this point.

Now imagine that you are a maintainer for Suse, and someone reports the same bug against a long-term service release.In practice, there are several different versions being maintained by different distributions, and no good way to know if the same bug has been reported and fixed by someone else. You end up searching for a fix in upstream bug trackers, and in the bug trackers of each of the other main distributions. According to Federico at the time:

Patches for old versions are traded in the black market. You have friends in another distro? You ask them first, “did you guys already fix this?” Those patches don’t ever manage to reach CVS, where everyone would be able to get them.

Ideally, you could collaborate ahead of time with other distributions to ensure that you are all using the same branch of upstream modules, and are committing patches upstream. The Linux kernel is moving to this model, and there are also discussions underway in GNOME to co-ordinate this type of activity. Mark Shuttleworth has also pushed for something similar by encouraging projects in the core Linux platform to have a regular cadence of releases, so that everyone can synchronise their longer term service offerings every couple of years.

But at the moment, the best you can hope for is that your patch will be included in an upcoming release for your distribution, and which point other users of the distro can avail of it, and that upstream will patch their development version and latest stable versions, and get your patch to everyone in a few months.

Working upstream

The goal of this article is to explain what working upstream actually means, and how to make that more palatable for a distribution that wants to get features written and included in their next release. Hopefully, by pointing out some of the shortcomings of the way patches circulate from developers to users, some of these issues can be addressed.

In any case, one thing is clear – if you are carrying a patch as a distribution without ever submitting it upstream, you are making a costly mistake. You will be carrying code that others won’t, and bearing all of the merge and maintenance burden for that code for years to come. The path to maximum happiness is to co-ordinate with other distributions and with upstream to ensure that everyone is working in the same place, and sharing work as much as possible.

Why the change of heart? My intention was never to make a fortune with the report, my main priority was covering my costs and time spent. And after 24 hours, I’ve achieved that. I have had several press requests for the full report, and requests from clients to be allowed to use the report both with press and with their clients.

This solution is the best for all involved, I think – I have covered my costs, the community (and everyone else) gets their hands on the report with analysis as soon as possible, and my clients are happy to have the report available under a license which allows them to use it freely.

You can download the full report now for free.

By looking at the modules in the GNOME 2.30 release, made last March, we aim to answer many of those questions, and give deeper insight into the motivations of participants in the project.

GNOME activity over time, horizontal bars are release dates

Here are our key findings:

• GNOME has a rhythm – there is a measurable increase in activity before release time, and after the annual GNOME conference GUADEC
• While over 70% of GNOME developers identify themselves as volunteers, over 70% of the commits to the GNOME releases are made by paid contributors
  

  GNOME committer self-identification - volunteer/professional

• Red Hat are the biggest contributor to the GNOME project and its core dependencies. Red Hat employees have made almost 17% of all commits we measured, and 11 of the top 20 GNOME committers of all time are current or past Red Hat employees. Novell and Collabora are also on the podium.
• A number of top company contributors are consultancy/services companies specialising in the GNOME platform – Collabora, CodeThink, Openismus, Lanedo and Fluendo are in the top 20 companies. As many of these companies grew initially through work on Maemo, this is a sign of the success of Nokia’s strategy around the GNOME stack.

One of the interesting things that we have done for the census is to look at who is maintaining modules by looking at commits over the past two years, and use this data to identify areas of the platform which see lots of collaboration, areas where the maintenance burden is left to volunteers, and areas where individual companies assume most of the maintenance burden.

There are a number of modules in the platform which see a considerable amount of co-opetition, including Evolution, Evolution Data Server, DBus and GStreamer. Most modules in the platform, however, are either maintained to a large extent by volunteer developers, or see the vast majority of their contributions from one company.

I see this information being useful for companies interested in using the GNOME platform for their products, companies seeking custom application development, potential large-scale customers of desktop Linux or customers buying high-level support who want to know who employs more module maintainers or committers to the project.

The GNOME maintenance map, with modules coloured according to the company maintaining them

Update: The release has now been published under a Creative Commons licence on October 1st 2010.

GNOME activity over time, horizontal bars are release dates

Who knew that educating people in simple sabotage (defined as sabotage not requiring in-depth training or materials) could have so much in common with communicating free software values? I read the OSS Simple Sabotage Field Manual (pdf) which has been doing the rounds of management and security blogs recently, and one article on “motivating saboteurs” caught my eye enough to share:

Personal Motives

• The ordinary citizen very probably has no immediate personal motive for committing simple sabotage. Instead, he must be made to anticipate indirect personal gain, such as might come with enemy evacuation or destruction of the ruling gov­ernment group. Gains should be stated as specifically as possible for the area addressed: simple sabotage will hasten the day when Commissioner X and his deputies Y and Z will be thrown out, when particu­larly obnoxious decrees and restrictions will be abolished, when food will arrive, and so on. Abstract verbalizations about personal liberty, freedom of the press, and so on, will not be convincing in most parts of the world. In many areas they will not even be comprehensible.
• Since the effect of his own acts is limited, the saboteur may become discouraged unless he feels that he is a member of a large, though unseen, group of saboteurs operating against the enemy or the government of his own country and elsewhere. This can be conveyed indirectly: suggestions which he reads and hears can include observations that a particular technique has been successful in this or that district. Even if the technique is not applicable to his surroundings, another’s success will encourage him to attempt similar acts. It also can be conveyed directly: statements praising the effectiveness of simple sabotage can be contrived which will be pub­lished by white radio, freedom stations, and the sub­versive press. Estimates of the proportion of the population engaged in sabotage can be disseminated. Instances of successful sabotage already are being broadcast by white radio and freedom stations, and this should be continued and expanded where com­patible with security.
• More important than (a) or (b) would be to create a situation in which the citizen-saboteur acquires a sense of responsibility and begins to educate others in simple sabotage.

Now doesn’t that sound familiar? Trying to convince people that free software is good for them because of the freedom doesn’t work directly – you need to tie the values of that freedom to something which is useful to them on a personal level.

“You get security fixes better because people can read the code”, “You have a wide range of support options for Linux because it’s free software and anyone can understand it”, “Sun may have been bought by Oracle, but you can continue to use the same products because anyone can modify the code, so others have taken up the maintenance, support and development burden”, and so on.

Providing (custom tailored) concrete benefits, which comes from freedom is the way to motivate people to value that freedom.

In addition, the point on motivation struck a cord – you need to make people feel like they belong, that their work means something, that they’re not alone and their effort counts, or they will become discouraged. A major job in any project is to make everyone feel like they’re driving towards a goal they have personally bought into.

Finally, you will only have succeeded when you have sufficiently empowered a saboteur to the point where they become an advocate themselves, and start training others in the fine arts – and this is a major challenge for free software projects too, where we often see people with willingness to do stuff, and have some difficulty getting them to the point where they have assimilated the project culture and are recruiting and empowering new contributors.

For those who haven’t read it yet, the document is well worth a look, especially the section on “General Interference with Organisations and Production”, which reads like a litany of common anti-patterns present in most large organisations; and if you never knew how to start a fire in a warehouse using a slow fuse made out of rope and grease, here’s your chance to find out.

Fernando Herrera, Claudio Saavedra, Alberto Garcia and myself will be running the two-day course, covering the basics of a Linux development environment and developer tools, the GNOME stack, including freedesktop.org components, and the social aspects of working with a free software project, being a good community citizen, getting your code upstream, and gaining influence in projects you work with.

The developer tools section will go beyond getting you compiling the software to also present mobile development environments, and the tools you can use to profile your apps, or diagnose I/O or memory issues, dealing with the vast majority of performance issues developers encounter.

This is the first time I have seen a training course which treats the soft science of working with free software communities, and given the number of times that people working in companies have told me that they need help in this area, I believe that this is satisfying a real need.

We are keeping the numbers down to ensure that the highest quality training & individual attention is provided – only 20 places are available. The pricing for the training course is very competitive for this type of course – €1500 per person, including training, meals and printed training materials, and a professional registration to GUADEC, worth €250.

If you register before June 15th, you can even get an additional discount – the early bird registration price is only €1200 per person.

I’m really excited about this, and I hope others will be too. This is the first time that we will have done training like this in conjunction with GUADEC, and I really hope that this will bring some new developers to the conference for the week, as well as being a valuable addition to the GUADEC event.

While this is true, it is also not the whole story. Trying to build a rock-solid software platform on shifting sands is not easy. Many projects do not commit to regular stable releases of their software. In the not too distant past, the video codecs produced by the MPlayer project, universally shipped in Linux distributions, had never had a stable or unstable release. The GIMP went from version 1.2.0 in December 1999 to 2.0.0 in March 2004 in unstable mode, with only bug-fix releases on the 1.2 series.

In these circumstances, getting both the stability your customers need, and the latest & greatest features, is not easy. Time-based releases, pioneered by the GNOME project in 2001, and now almost universally followed by major free software projects, mitigate this. They give you periodic sync points where you can get software which meets a certain standard of feature stability and robustness. But no software release is bug-free, and this is true for both free and proprietary software. In the Mythical Man-Month, Fred Brooks described the difficulties of system integration, and estimated that 25% of the time in a project would be spent integrating and testing relationships between components which had already been planned, written and debugged. Building a system or a Linux distribution, then, takes a lot longer than just throwing the latest stable version of every project together and hoping it all works.

By participating actively in the QA process of the project leading up to the release, and by maintaining automated test suites and continuous integration, you can mitigate the effects of both the shifting sands of unstable development versions and reduce the integration overhead once you have a stable release.At some stage, you must draw a line in the sand, and start preparing for a release. In the GNOME project, we have a progressive freezing of modules, progressively freezing the API & ABI of the platform, the features to be included in existing modules, new module proposals, strings and user interface changes, before finally we have a complete code freeze pre-release. Similarly, distributors decide early what versions of components they will include on their platforms, and while occasional slippages may be tolerated, moving to a newmajor version of a major component of the platform would cause integration testing to return more or less to zero – the overhead is enormous.

The difficulty, then, is what to do once this line is drawn. Serious bugs will be fixed in the stable branch, and they can be merged into your platform easily. But what about features you develop to solve problems specific to your device? Typically, free software projects expect new features to be built and tested on the unstable branch, but you are building your platform on the stable version. You have three choices at this point, none pleasant – never merge, merge later, or merge now:

• Develop the feature you want on your copy of the stable branch, resulting in a delta which will be unique to your code-base, which you will have to maintain separately forever. In addition, if you want to benefit from the features and bug fixes added to later versions of the component, you will incur the cost of merging your changes into the latest version, a non-negigible amount of time.
• Once you have released your product and your team has more time, propose the features you have worked on piecemeal to the upstream project, for inclusion in the next stable version. This solution has many issues:
  • If the period is long enough, your feature additions will be long removed from the codebase as it has evolved, and merging your changes into the latest unstable tree will be a major task
  • You may be redundantly solving problems that the community has already addressed, in a different or incompatible way.
  • Feature requests may need substantial re-writing to meet community standards. This problem is doubly so if you have not consulted the community before developing the feature, to see how it might best be integrated.
  • In the worst case, you may have built a lot of software on an API which is only present in your copy of the component’s source tree, and if your features are rejected, you are stuck maintaining the component, or re-writing substantial amounts of code to work with upstream.
• Develop your feature on the unstable branch of the project, submit it for inclusion (with the overhead that implies), and back-port the feature to your stable branch once included. This guarantees a smaller delta from the next stable version to your branch, and ensures you work gets upstream as soon as possible, but adds a time & labour overhead to the creation of your software platform

In all of these situations there is a cost. The time & effort of developing software within the community and back-porting, the maintenance cost (and related unleveraged potential) to maintaining your own branch of a major component, and the huge cost of integrating a large delta back to the community-maintained version many months after the code has been written.

Intuitively, it feels like the long-term cheapest solution is to develop, where possible, features in the community-maintained unstable branch, and back-port them to your stable tree when you are finished. While this might be nice in an ideal world, feature proposals have taken literally years to get to the point where they have been accepted into the Linux kernel, and you have a product to ship – sometimes the only choice you have is to maintain the feature yourself out-of-tree, as Robert Love did for over a year with inotify.

While addressing the raw value of the code produced by the community in the interim, Mal does not quantify the costs associated with these options. Indeed, it is difficult to do so. In some cases, there is not only a cost in terms of time & effort, but also in terms of goodwill and standing of your engineers within the community – this is the type of cost which it is very hard to put a dollar value on. I would like to see a way to do so, though, and I think that it would be possible to quantify, for example, the community overhead (as a mean) by looking at the average time for patch acceptance and/or number of lines modified from intial proposal to final mainline merge.

Anyone have any other thoughts on ways you could measure the cost of maintaining a big diff, or the cost of merging a lot of code?

Governance comes in many shapes & sizes of course. My favourite governance stories are about federating individuals, who manage to channel community efforts, maintain a meritocracy where code talks, and yet don’t come across as authoritarians.

Outside of Linus (who’s a good example), Ton Roosendaal of Blender has this kind of presence. Talking to Ton, it is easy to see that he cares about Blender and about the Blender Community. The care and attention that he brings to projects like “Elephants Dream” and “Big Buck Bunny”, or to the supporting documentation and conferences he organises for the community, illustrate the esteem in which he holds his users and his developer community. Even the way the Blender Foundation came into being was amazing.

One of my favourite communities is Inkscape. When they broke from Sodipodi, there was this acrimonious flame war, and something of a bitter taste in people’s mouth. So what Bryce Harrington, Nathan Hurst, MenTaLguY and Ted Gould did when they split was decide to throw open the doors, and accept code from all comers. They set a direction and some ambitious goals, but they were very clear from the start – come right in, you’re welcome. And this gave the project some great results, especially early on when it was still establishing itself. Bryce describes one of them in this article.

The success of the Inkscape project’s governance model is borne out by its ability to escape founder’s syndrome – Bryce, Nathan and Ted have now backed away from the project to some extent, they’re still there as wise heads, but they have passed off the direction of the project to other capable people.

I think the way that Drizzle was born bears some resemblance to this, and I really like the way they have consciously broken down the walls which were necessarily up around MySQL. Brian Aker’s been something of an inspiration on this. His mission statement at the announcement of the project was astounding.

Subversion‘s governance model is an exemplar of best practices too. Set a clear project scope (“Subversion is a compelling alternative to CVS”), clear goals, establish transparent and fair community processes, and open up the gates. Anything within the scope of the project is fair game. And once again, code talks. This story, from Karl Fogel’s “Producing OSS” illustrated the robustness of their governance, and the confidence the project’s leaders had in their ability to influence the project.

The Maemo Community Council has the potential to be a very good governance structure, I think.  The idea of a governing body of the community, by the community, for the community, whose goal is to canalise the efforts of a disparate group into something coherent, and to provide a legitimate point f contact for technical decision-makers in Nokia, is a novel one, and hasn’t been tried, as far as I can tell, by other companies.

Counter-examples of good governance are all around, I won’t name any in particular to protect the guilty. But many of them stem from a misguided belief in absolute free speech, to the detriment of the quality of discourse and code in the project (“we are all created equal”) which results in very chatty, but unproductive, individuals taking senior positions in the community, or a sort of shyness of the founder or leader, who doesn’t believe that it’s his place to set a direction and tone.In company-run projects, excessive control or influence is an equally toxic characteristic. Companies who retain a veto on community decisions are companies who do not trust their communities.

•  From the archives: the best distros of 2000 | TuxRadar: A trip down Linux distribution memory lane – back to the day when WindowMaker was considered “an attractive alternative” to Enlightenment, the old default GNOME window manager.
• Polymorph: Hacking Business Models: A few months ago, Monty Widenius and Zack Greant got together to talk about what a really great company might look like. Now that he has left Sun and MySQL, Monty is going to try to put the theory into practice.
• TG Daily – Linux saga: Girl drops out of school over Ubuntu: The zealots ruin an otherwise great story on how a girl mistakenly got delivered an Ubuntu laptop, had trouble fulfilling her course requirements and connecting to her broadband supplier, and had all her problems solved by a local TV station. Rather than the story being about the problems the girl encountered – a key opportunity to educate people about Linux – it is now about how rude and insulting Linux supporters can be. Talk about snatching defeat from the jaws of victory.
• There is no Open Source Community: Building Vibrant Open Source Communities: An interesting presentation on building community from OpenCollabNet community manager John Mark Walker
• Herding cats for fun and profit: Four tips for working with online communities | Community, Incorporated | ZDNet.com: Joe “Zonker” Brockmeier, OpenSuse’s community manager, gives his top community management tips
• The Art of Community: Jono Bacon of Ubuntu has been commissioned to write a book on building community by O’Reilly, and as a case in point is writingthe book as a community effort, inviting guest authors and feedback all the way

Moving the Mobile industry from purchasing to co-development in free software communities

Recently, Matt Aslett wrote an article about the way that attitudes to free software evolve over time within a company, using a graphic he got from the Eclipse Foundation, based on some Nortel funded research. Software sneaks in on the ground floor, going from simple use of components to a real understanding of community-driven development, resulting, long-term, in building free software projects and strategies.

Matt sees an evolution in attitudes as the software and its value is discovered at different levels of the organisation, before finally the business development side of the company picks up the ball and drives free software into the heart of the company’s product strategy.

I have also seen this learning process in action, but I would express it differently. People discover the value of the freedoms granted by free software one by one, more or less independently of their level in an organisation – exploring each freedom before discovering its limitations, and thus discovering the value of the next freedom, and qualifying for the next level.

The core freedoms in the Free Software Definition which are granted to the user of free software are:

1. Freedom to use
2. Freedom to modify
3. Freedom to share, freedom to redistribute
4. Freedom to participate

As companies start to integrate free software components into their products, they discover the value of these freedoms one by one.

Use

The first thing that people see about free software is FREE! As in zero cost. The days when companies reject a product out of hand because they don’t have to pay for it are gone – Linux, OpenOffice.org, Apache, Red Hat and a plethora of other “free” products have proven themselves in the marketplace, and companies are now prepared to allow free software components into their solutions, after appropriate consideration of the licences involved.

To quote one attendee at MAPOS 08, “why would I want to write a compression library, when I can download the best one in the world from zlib.org?” In the area of specialised components for secure communications, compression/decompression, a commodity kernel, and a bunch of other situations, it is appropriate to use free software components off-the-shelf. We expect them to work, and we don’t expect to ever need to talk to the maintainer.

Free software components are in use like this in thousands of systems solutions and commercial products, often without their authors even being aware of it. The main advantage of this for a systems or product company is a saving of time and money, through having a fully functional component without having to go through a purchasing process, and a reduced software bill of materials. An additional advantage is the simplification of your licensing due diligence, thanks to the relatively well-understood consequences of the various popular free software licences.

The difficulty arises when the software doesn’t meet your needs. In many cases, libraries are written by an individual to scratch an itch – it works for him, but is not quite up to your requirements. As one friend of mine put it: “Open Source: 80% as good as the last guy needed it to be”.

Perhaps it’s software that works on 32 bit platforms, but has never been tested for 64 bit. Perhaps it has not been ported to ARM or MIPS. Or perhaps the author simply never imagined that anyone would want the feature which you find indispensable.

In this situation, you can always ask the software author to write the feature or fix the bug for you – but since there is no client/supplier relationship between you, it is entirely reasonable for a volunteer to put your request on the long finger, or reject it outright.

At this point, you realise the value of having the source code – you can modify the software to meet your needs, or pay someone else to do it for you.

Modify

Being able to modify software that doesn’t quite meet your needs is amazing. This is the way things used to work by default, but the shrink-wrapped software revolution of the 1980s got everyone used to the idea that software was a valuable asset to be protected from public view at all costs. When I worked for Informix in the late ’90s, we used to refer to the source code of our leading product as “the crown jewels”.

With the widespread acceptance of free software as an alternative, developers are no longer surprised when they may see how a program works, and change its behaviour. This ability brings two important and immediate benefits – you have control of the behaviour of the software, and you can adapt it to suit exactly your needs. The old choice of build vs buy has become: build vs buy vs extend.

This situation is common in software services companies which provide vertically integrated “solutions” to corporate clients. You take components where you can find them to speed up initial development, stick everything together with duct-tape, hack whatever you need in whatever libraries you’re using to make everything pass the client’s integration tests, and then publish a set of .tar.gz files somewhere on the website of the company to fulfil any licensing requirements.

This control and ability to tailor a solution comes at a price, however. Over and above the cost of making the changes, your team is lumbered with a maintenance problem. Let’s say that implementing the features you need on top of a component the first time round takes a month. Fixing bugs in the features when it has been rolled out can take another few weeks. A few months later, the upstream product you’re based on goes and releases a shiny new version, with lots of compelling new features that you really want.

The cost of integrating your features into the newer version, and doing extensive regression testing before rolling out the new version, might take you another 6 weeks. It is not unusual for time spent integrating your work into later versions to quickly outweigh initial development time and investment. Inconveniently, this is typically effort which is not budgeted for beforehand.

After a company has run into this problem a couple of times, over the course of a year or two, someone will usually suggest that you propose that the features you have developed be sent upstream to the projects you work with – if the feature is accepted, you have solved your maintenance problem, it will be in all future releases of the project, and all of that tricky integration work and regression testing work will get done upstream, as part of normal maintenance.

Redistribute

And so you tell your star hacker Jack that he has two weeks to get your 5,000 line patch down to manageable size by getting your work integrated upstream. (when I said this at MAPOS, no-one laughed – so maybe this does not sound as ridiculous as I thought it did).

He diligently goes to work, cleaning up his code, getting rid of all the warnings, splitting up the big diff into small manageable chunks, creating accounts in 10 different bug trackers, signing up to a dozen mailing lists, creating 47 bugs with terse descriptions, attaching proposed bug fixes, and for major features he sends email telling people that the feature is there and asking for review.

By the end of a frantic month, two weeks more than he was given, he reckons that if everything he’s submitted is accepted, your 5,000 patch will be down to a more manageable 2,000 line patch.

What happens next is… underwhelming.

Major features and bug fixes lie unreviewed for weeks or months. Those that are reviewed need changes which take time and effort. Some patches are rejected outright because they’re too big and the feature is difficult to review.

A post mortem analysis of the project of “giving back to the community” might identify some of the following conclusions:

• Not enough time and resources were devoted to advocating your changes upstream
• Personal relationships between Jack and the project maintainers led to a much higher acceptance rate for patches and feature requests
• The projects were initially evaluated on technical grounds, no thought was given to the developer community underpinning it
• In some cases, maintainers priorities were ill-understood

There are two common conclusions that people make from this kind of analysis;

1. It’s not worth it. They don’t want our work, and the time we’re spending is costing us more than maintaining out-of-tree patches
2. Perhaps if you had engaged with the projects before modifying them heavily, or had been regularly sending contributions, that the maintainers would have been more encouraging, and might have been more prepared to consider your work. If someone from your company was a maintainer or committer already, you would have had a valuable short-cut to getting your agenda implemented in the upstream project.

If you choose door number 1, you will go no further in your quest to really understanding free software processes. This is a reasonable thing to do, but the costs involved are often miscalculated. In addition, the benefits of influencing upstream projects are often vastly underestimated.

If you choose door number 2, you have concluded, in short, that it is madness to include a component in one of your products and exert no influence with upstream projects.

Participate

To have influence, you must understand how the community around a project works. Someone within the team must become an active, trusted member of the community. Once they have gained the trust of the community through their contributions, there may be some procedure to follow for them to become a maintainer of the project, or to gain commit privileges.

These considerations are not technical, for the most part. Friendship and trust are fuzzy human concepts. And this more than anything else brings me to my final point.

Community is hard

For a start, every community is different. They all have different people, different behavioural norms, different dynamics, different forums for communication.

Taking GNOME Mobile as an example, there are 18 projects in the GNOME Mobile platform, with another 10 or so in incubation. Within that, we have a large number of projects housed on gnome.org, and governed by our rules, procedures and conventions. And yet each project has its own set of maintainers – GTK+ is maintained by a committee of around 10 people, EDS is maintained principally by Novell employees, gtkmm has one core maintainer, and so on.

On top of this are a number of freedesktop.org projects, and a couple more which are not under either of these umbrellas. To be an effective influencer of GNOME Mobile, you need to learn the culture of over 20 projects, of wildly varying sizes and baggage.

There are a number of issues to bear in mind when you approach a free software community for the first time. The main one is that while the vast majority of projects think that they are welcoming people with open arms, if you are a stranger to their land, it is very likely that you will be getting exactly the opposite message.

In some cases, the extent of the welcome is “go and read wiki page telling people how to contribute to the project”. In other cases, no wiki page exists. Occasionally, you will be told that you’re asking your question on the wrong mailing list, or in the wrong way, or that you should read the relevant documentation first. It is not unusual for people to answer questions with a very terse answer – perhaps a link to a mailing list discussion or web-page where the answer can be found.

In general, all of these things are intended to fulfil a simple goal – get you the information you want as quickly as possible, in a way that wastes the time of people already in the project as little as possible. An admirable goal indeed, but as a newcomer, this is not how people are used to being welcomed. Eric Raymond wrote extensively about this in his essay “How to ask questions the smart way”.

Indeed, one of the hardest things to do as an outsider looking in is to evaluate when a community is healthy and viable, and when it has problems which will prevent you from working effectively in partnership. Few resources which talk about healthy free software community projects exist – “Producing Open Source Software”, by Karl Fogel, is something of a bible on the subject, and should be required reading for anyone considering investing in free software. I have also found some presentations, including Simon Phipps’s 2006 OSCon keynote “The Zen of Free” and “How Open Source Projects Survive Poisonous People” by Ben Collins-Sussman and Brian Fitzpatrick, to be excellent resources in helping identify traits of what makes up a healthy community. Two other useful papers which include metrics on measuring the openness of a community, including its governance model, are Pia Waugh’s “The Foundations of Openness” and François Druel’s Ph.D. Thesis (in French) “Evaluation de la valeur à l’ère du Web” (PDF – rough translation: “Measuring value in the era of the Web”).

Some of the considerations when evaluating a community are whether there is clear leadership, whether that leadership is an individual, a group, or a company, how the leaders are chosen (if they are chosen), what technological and social barriers to participating in the project exist, whether the community processes are documented and transparent, what recourse one has if one feels badly treated, what the behavioural norms of the community are (and whether they are documented) – the list goes on. Pia’s paper in particular gives a great overview in the section “Open Governance”.

Call to arms

And so I close with a call to arms to both free software communities, and companies planning on developing an “open source strategy”.

First, developers, document your communities. Think of yourselves as guides, explaining the cultural quirks of your country to a newly arrived immigrant. Be explicit. In addition to explaining where and how your community works, document how one gains trust and responsibility. Ensure that a newcomer can learn quickly what he needs to do to become a citizen and from there a project maintainer. I am not saying that it should be easy for someone to become a maintainer. What I am suggesting is that it should be easy to see how one becomes a maintainer before doing it.

Next, project managers, software developers, company leaders: please, please, please – save yourself time and money and, when you reach the point where you will be building products which depend on good free software components, let the second thing that you do, right after a technical evaluation, be to evaluate the health of the community. A community where you can earn influence and guide the project to better meet your needs is a better long-term investment than betting on a slightly technically superior solution with an unhealthy governance model.

You are building products that you will be selling, supporting, and hopefully profiting from. In this situation, does it really make sense not to have the developer’s ear?