While this is true, it is also not the whole story. Trying to build a rock-solid software platform on shifting sands is not easy. Many projects do not commit to regular stable releases of their software. In the not too distant past, the video codecs produced by the MPlayer project, universally shipped in Linux distributions, had never had a stable or unstable release. The GIMP went from version 1.2.0 in December 1999 to 2.0.0 in March 2004 in unstable mode, with only bug-fix releases on the 1.2 series.

In these circumstances, getting both the stability your customers need, and the latest & greatest features, is not easy. Time-based releases, pioneered by the GNOME project in 2001, and now almost universally followed by major free software projects, mitigate this. They give you periodic sync points where you can get software which meets a certain standard of feature stability and robustness. But no software release is bug-free, and this is true for both free and proprietary software. In the Mythical Man-Month, Fred Brooks described the difficulties of system integration, and estimated that 25% of the time in a project would be spent integrating and testing relationships between components which had already been planned, written and debugged. Building a system or a Linux distribution, then, takes a lot longer than just throwing the latest stable version of every project together and hoping it all works.

By participating actively in the QA process of the project leading up to the release, and by maintaining automated test suites and continuous integration, you can mitigate the effects of both the shifting sands of unstable development versions and reduce the integration overhead once you have a stable release.At some stage, you must draw a line in the sand, and start preparing for a release. In the GNOME project, we have a progressive freezing of modules, progressively freezing the API & ABI of the platform, the features to be included in existing modules, new module proposals, strings and user interface changes, before finally we have a complete code freeze pre-release. Similarly, distributors decide early what versions of components they will include on their platforms, and while occasional slippages may be tolerated, moving to a newmajor version of a major component of the platform would cause integration testing to return more or less to zero – the overhead is enormous.

The difficulty, then, is what to do once this line is drawn. Serious bugs will be fixed in the stable branch, and they can be merged into your platform easily. But what about features you develop to solve problems specific to your device? Typically, free software projects expect new features to be built and tested on the unstable branch, but you are building your platform on the stable version. You have three choices at this point, none pleasant – never merge, merge later, or merge now:

• Develop the feature you want on your copy of the stable branch, resulting in a delta which will be unique to your code-base, which you will have to maintain separately forever. In addition, if you want to benefit from the features and bug fixes added to later versions of the component, you will incur the cost of merging your changes into the latest version, a non-negigible amount of time.
• Once you have released your product and your team has more time, propose the features you have worked on piecemeal to the upstream project, for inclusion in the next stable version. This solution has many issues:
  • If the period is long enough, your feature additions will be long removed from the codebase as it has evolved, and merging your changes into the latest unstable tree will be a major task
  • You may be redundantly solving problems that the community has already addressed, in a different or incompatible way.
  • Feature requests may need substantial re-writing to meet community standards. This problem is doubly so if you have not consulted the community before developing the feature, to see how it might best be integrated.
  • In the worst case, you may have built a lot of software on an API which is only present in your copy of the component’s source tree, and if your features are rejected, you are stuck maintaining the component, or re-writing substantial amounts of code to work with upstream.
• Develop your feature on the unstable branch of the project, submit it for inclusion (with the overhead that implies), and back-port the feature to your stable branch once included. This guarantees a smaller delta from the next stable version to your branch, and ensures you work gets upstream as soon as possible, but adds a time & labour overhead to the creation of your software platform

In all of these situations there is a cost. The time & effort of developing software within the community and back-porting, the maintenance cost (and related unleveraged potential) to maintaining your own branch of a major component, and the huge cost of integrating a large delta back to the community-maintained version many months after the code has been written.

Intuitively, it feels like the long-term cheapest solution is to develop, where possible, features in the community-maintained unstable branch, and back-port them to your stable tree when you are finished. While this might be nice in an ideal world, feature proposals have taken literally years to get to the point where they have been accepted into the Linux kernel, and you have a product to ship – sometimes the only choice you have is to maintain the feature yourself out-of-tree, as Robert Love did for over a year with inotify.

While addressing the raw value of the code produced by the community in the interim, Mal does not quantify the costs associated with these options. Indeed, it is difficult to do so. In some cases, there is not only a cost in terms of time & effort, but also in terms of goodwill and standing of your engineers within the community – this is the type of cost which it is very hard to put a dollar value on. I would like to see a way to do so, though, and I think that it would be possible to quantify, for example, the community overhead (as a mean) by looking at the average time for patch acceptance and/or number of lines modified from intial proposal to final mainline merge.

Anyone have any other thoughts on ways you could measure the cost of maintaining a big diff, or the cost of merging a lot of code?

The false assumption here is that only software companies can make money off free software in a “pure” way.

There are lots of ways of making money leveraging free software. Simon Phipps writes often on about the core principle behind some of these.

First, there are all the hardware-based models. Reduce your per-unit costs for some hardware you’re manufacturing because you’re integrating a free software stack. Make a better product, because you can modify the software running on it from top to bottom. Sell more server hardware because your software is more widely distributed. To paraphrase Doc Searls, people aren’t making money with free software, they’re making money because of free software.

Second, you have the software as a service business model. If your software gets more useful the more people use it, then at some stage, it doesn’t matter whether it’s free software or not – the value is in the user base and the data. Imagine a SugarCRM where there were substantial network effects in having an account on their hosted service. Eventually, it wouldn’t matter that the platform is available for download, the value would be in being hosted.

And thirdly, you have what Simon says “payment at the point of value”. Get your software out there. Make sure that you’re heavily deployed. And set yourself up so that when people need help they come to you. This is the “pure open source” play, and it is the one which I have talked about before. In this model, you’re open to competition, since you are monetising a value-adding support and expertise in a product which anyone can master and change.

There is a fourth way of making money off free software which doesn’t make sense for a company, but might make a lot of sense for an individual developer. And that is, get paid to do what you love by someone who belongs to the first three groups, and who values your expertise. Thus, Nokia, Novell, Red Hat  and Canonical hire established  free software developers to keep on doing what they were doing before, for money – because what they were doing has value to those companies in the context of their business model. Companies such as these, and other companies who hire people to work on products they depend on, are ensuring the future of free software.

If your company is building products on free software projects, then you would be a fool not to invest in those projects to ensure they continue to develop and improve. You can invest with a service contract, by outsourcing to a company like Spikesource, or by hiring in a hacker – the net result is the same. Companies using the software will ensure that key free software projects will continue to develop, independent of any one business model.

Moving the Mobile industry from purchasing to co-development in free software communities

Recently, Matt Aslett wrote an article about the way that attitudes to free software evolve over time within a company, using a graphic he got from the Eclipse Foundation, based on some Nortel funded research. Software sneaks in on the ground floor, going from simple use of components to a real understanding of community-driven development, resulting, long-term, in building free software projects and strategies.

Matt sees an evolution in attitudes as the software and its value is discovered at different levels of the organisation, before finally the business development side of the company picks up the ball and drives free software into the heart of the company’s product strategy.

I have also seen this learning process in action, but I would express it differently. People discover the value of the freedoms granted by free software one by one, more or less independently of their level in an organisation – exploring each freedom before discovering its limitations, and thus discovering the value of the next freedom, and qualifying for the next level.

The core freedoms  in the Free Software Definition which are granted to the user of free software are:

1. Freedom to use
2. Freedom to modify
3. Freedom to share, freedom to redistribute
4. Freedom to participate

As companies start to integrate free software components into their products, they discover the value of these freedoms one by one.

Use

The first thing that people see about free software is FREE! As in zero cost. The days when companies reject a product out of hand because they don’t have to pay for it are gone – Linux, OpenOffice.org, Apache, Red Hat and a plethora of other “free” products have proven themselves in the marketplace, and companies are now prepared to allow free software components into their solutions, after appropriate consideration of the licences involved.

To quote one attendee at MAPOS 08, “why would I want to write a compression library, when I can download the best one in the world from zlib.org?” In the area of specialised components for secure communications, compression/decompression, a commodity kernel, and a bunch of other situations, it is appropriate to use free software components off-the-shelf. We expect them to work, and we don’t expect to ever need to talk to the maintainer.

Free software components are in use like this in thousands of systems solutions and commercial products, often without their authors even being aware of it. The main advantage of this for a systems or product company is a saving of time and money, through having a fully functional component without having to go through a purchasing process, and a reduced software bill of materials. An additional advantage is the simplification of your licensing due diligence, thanks to the relatively well-understood consequences of the various popular free software licences.

The difficulty arises when the software doesn’t meet your needs. In many cases, libraries are written by an individual to scratch an itch – it works for him, but is not quite up to your requirements. As one friend of mine put it: “Open Source: 80% as good as the last guy needed it to be”.

Perhaps it’s software that works on 32 bit platforms, but has never been tested for 64 bit. Perhaps it has not been ported to ARM or MIPS. Or perhaps the author simply never imagined that anyone would want the feature which you find indispensable.

In this situation, you can always ask the software author to write the feature or fix the bug for you – but since there is no client/supplier relationship between you, it is entirely reasonable for a volunteer to put your request on the long finger, or reject it outright.

At this point, you realise the value of having the source code – you can modify the software to meet your needs, or pay someone else to do it for you.

Modify

Being able to modify software that doesn’t quite meet your needs is amazing. This is the way things used to work by default, but the shrink-wrapped software revolution of the 1980s got everyone used to the idea that software was a valuable asset to be protected from public view at all costs. When I worked for Informix in the late ’90s, we used to refer to the source code of our leading product as “the crown jewels”.

With the widespread acceptance of free software as an alternative, developers are no longer surprised when they may see how a program works, and change its behaviour. This ability brings two important and immediate benefits – you have control of the behaviour of the software, and you can adapt it to suit exactly your needs. The old choice of build vs buy has become: build vs buy vs extend.

This situation is common in software services companies which provide vertically integrated “solutions” to corporate clients. You take components where you can find them to speed up initial development, stick everything together with duct-tape, hack whatever you need in whatever libraries you’re using to make everything pass the client’s integration tests, and then publish a set of .tar.gz files somewhere on the website of the company to fulfil any licensing requirements.

This control and ability to tailor a solution comes at a price, however. Over and above the cost of making the changes, your team is lumbered with a maintenance problem. Let’s say that implementing the features you need on top of a component the first time round takes a month. Fixing bugs in the features when it has been rolled out can take another few weeks. A few months later, the upstream product you’re based on goes and releases a shiny new version, with lots of compelling new features that you really want.

The cost of integrating your features into the newer version, and doing extensive regression testing before rolling out the new version, might take you another 6 weeks. It is not unusual for time spent integrating your work into later versions to quickly outweigh initial development time and investment. Inconveniently, this is typically effort which is not budgeted for beforehand.

After a company has run into this problem a couple of times, over the course of a year or two, someone will usually suggest that you propose that the features you have developed be sent upstream to the projects you work with – if the feature is accepted, you have solved your maintenance problem, it will be in all future releases of the project, and all of that tricky integration work and regression testing work will get done upstream, as part of normal maintenance.

Redistribute

And so you tell your star hacker Jack that he has two weeks to get your 5,000 line patch down to manageable size by getting your work integrated upstream. (when I said this at MAPOS, no-one laughed – so maybe this does not sound as ridiculous as I thought it did).

He diligently goes to work, cleaning up his code, getting rid of all the warnings, spliting up the big diff into small manageable chunks, creating accounts in 10 different bug trackers, signing up to a dozen mailing lists, creating 47 bugs with terse descriptions, attaching proposed bug fixes, and for major features he sends email telling people that the feature is there and asking for review.

By the end of a frantic month, two weeks more than he was given, he reckons that if everything he’s submitted is accepted, your 5,000 patch will be down to a more manageable 2,000 line patch.

What happens next is… underwhelming.

Major features and bug fixes lie unreviewed for weeks or months. Those that are reviewed need changes which take time and effort. Some patches are rejected outright because they’re too big and the feature is difficult to review.

A post mortem analysis of the project of “giving back to the community” might identify some of the following conclusions:

• Not enough time and resources were devoted to advocating your changes upstream
• Personal relationships between Jack and the project maintainers led to a much higher acceptance rate for patches and feature requests
• The projects were initially evaluated on technical grounds, no thought was given to the developer community underpinning it
• In some cases, maintainers priorities were ill-understood

There are two common conclusions that people make from this kind of analysis;

1. It’s not worth it. They don’t want our work, and the time we’re spending is costing us more than maintaining out-of-tree patches
2. Perhaps if you had engaged with the projects before modifying them heavily, or had been regularly sending contributions, that the maintainers would have been more encouraging, and might have been more prepared to consider your work. If someone from your company was a maintainer or committer already, you would have had a valuable short-cut to getting your agenda implemented in the upstream project.

If you choose door number 1, you will go no further in your quest to really understanding free software processes. This is a reasonable thing to do, but the costs involved are often miscalculated. In addition, the benefits of influencing upstream projects are often vastly underestimated.

If you choose door number 2, you have concluded, in short, that it is madness to include a component in one of your products and exert no influence with upstream projects.

Participate

To have influence, you must understand how the community around a project works. Someone within the team must become an active, trusted member of the community. Once they have gained the trust of the community through their contributions, there may be some procedure to follow for them to become a maintainer of the project, or to gain commit privileges.

These considerations are not technical, for the most part. Friendship and trust are fuzzy human concepts. And this more than anything else brings me to my final point.

Community is hard

For a start, every community is different. They all have different people, different behavioural norms, different dynamics, different forums for communication.

Taking GNOME Mobile as an example, there are 18 projects in the GNOME Mobile platform, with another 10 or so in incubation. Within that, we have a large number of projects housed on gnome.org, and governed by our rules, procedures and conventions. And yet each project has its own set of maintainers – GTK+ is maintained by a committee of around 10 people, EDS is maintained principally by Novell employees, gtkmm has one core maintainer, and so on.

On top of this are a number of freedesktop.org projects, and a couple more which are not under either of these umbrellas. To be an effective influencer of GNOME Mobile, you need to learn the culture of over 20 projects, of wildly varying sizes and baggage.

There are a number of issues to bear in mind when you approach a free software community for the first time. The main one is that while the vast majority of projects think that they are welcoming people with open arms and are very welcoming, if you are a stranger to their land, it is very likely that you will be getting exactly the opposite message.

In some cases, the extent of the welcome is “go and read wiki page telling people how to contribute to the project”. In other cases, no wiki page exists. Occasionally, you will be told that you’re asking your question on the wrong mailing list, or in the wrong way, or that you should read the relevant documentation first. It is not unusual for people to answer questions with a very terse answer – perhaps a link to a mailing list discussion or web-page where the answer can be found.

In general, all of these things are intended to fulfil a simple goal – get you the information you want as quickly as possible, in a way that wastes the time of people already in the project as little as possible. An admirable goal indeed, but as a newcomer, this is not how people are used to being welcomed. Eric Raymond wrote extensively about this in his essay “How to ask questions the smart way”.

Indeed, one of the hardest things to do as an outsider looking in is to evaluate when a community is healthy and viable, and when it has problems which will prevent you from working effectively in partnership. Few resources which talk about healthy free software community projects exist – “Producing Open Source Software”, by Karl Fogel, is something of a bible on the subject, and should be required reading for anyone considering investing in free software. I have also found some presentations, including Simon Phipps’s 2006 OSCon keynote “The Zen of Free” and “How Open Source Projects Survive Poisonous People” by Ben Collins-Sussman and Brian Fitzpatrick, to be excellent resources in helping identify traits of what makes up a healthy community. Two other useful papers which include metrics on measuring the openness of a community, including its governance model, are Pia Waugh’s “The Foundations of Openness” and François Druel’s Ph.D. Thesis (in French) “Évaluation de la valeur à l’ère du Web ” (PDF – rough translation: “Measuring value in the era of the Web”).

Some of the considerations when evaluating a community are whether there is clear leadership, whether that leadership is an individual, a group, or a company, how the leaders are chosen (if they are chosen), what technological and social barriers to participating in the project exist, whether the community processes are documented and transparent, what recourse one has if one feels badly treated, what the behavioural norms of the community are (and whether they are documented) – the list goes on. Pia’s paper in particular gives a great overview in the section “Open Governance”.

Call to arms

And so I close with a call to arms to both free software communities, and companies planning on developing an “open source strategy”.

First, developers, document your communities. Think of yourselves as guides, explaining the cultural quirks of your country to a newly arrived immigrant. Be explicit. In addition to explaining where and how your community works, document how one gains trust and responsibility. Ensure that a newcomer can learn quickly what he needs to do to become a citizen and from there a project maintainer. I am not saying that it should be easy for someone to become a maintainer. What I am suggesting is that it should be easy to see how one becomes a maintainer before doing it

Next, project managers, software developers, company leaders: please, please, please – save yourself time and money and, when you reach the point where you will be building products which depend on good free software components, let the second thing that you do, right after a technical evaluation, be to evaluate the health of the community. A community where you can earn influence and guide the project to better meet your needs is a better long-term investment than betting on a slightly technically superior solution with an unhealthy governance model.

You are building products that you will be selling, supporting, and hopefully profiting from. In this situation, does it really make sense not to have the developer’s ear?

Dave Neary is reading all kinds of maemo content as we speak: developer documentation, mailing list archives, web pages at maemo.org… He has a good reason for that: Nokia is funding some of his time to work as the content guy for the maemo community. Just like Niels Breet, he is being paid to follow your agenda and move forward those tasks and topics that interest you more.

Dave is a good organizer, communicator, writer and DIY professional, with proven tracks in the GIMP, GNOME and OpenWengo communities. While Niels concentrates on the web applications and server infrastructure, Dave takes care of the content in the website, and the ways for you to help improving it and get the ownership. Last year we made some steps in this direction with the social news and the revamped downloads. Now the revamped wiki plan and the fact of having someone listening and thinking how to go further should bring us to a next level.

I am looking forward to working with the maemo community and Nokia to make their collaboration rock.

The GNOME Mobile Initiative, which first met at GUADEC, the GNOME Users’ and Developers’ European Conference in 2006 and publicly launched in April 2007, is a community effort to ensure that free and open source software is optimized for the growing Linux-based mobile device space. The Initiative has already had several meetings, both formally and
informally, and garnered considerable community and industry support.

Dave Neary, the co-ordinator of the GNOME Mobile track at the summit, believes that this meeting will accelerate the adoption of the GNOME platform on mobile devices. “Members of the GNOME Mobile group have been realising the leverage that collaborating closely with a free software community can give. Improved time to market, reduced R&D and maintenance costs, and above all, a highly performant and capable application platform on which to build your applications.”

Up to this point, the focus has been on co-ordinating integration efforts and reducing the amount of code being maintained outside the project, but that focus is expected to change as the initiative continues to mature and grow.

Ross Burton of OpenedHand, recently appointed release manager of the initiative, outlines his plans for the future of the project: “We are now moving beyond the initial phase of co-operation which consisted in people centralising work which they had been doing inside their companies to the core products. The next step is a roadmap which will systematically address the needs of consumers of the GNOME Mobile platform and ensure that the work is done in the community, and the creation of a mobile-specific release set of GNOME and GNOME-related projects.”

GNOME Mobile has a growing number of members, including industry heavyweights such as Nokia, ACCESS and FIC, the support of mobile consortia LiPS, the Linux Foundation’s MLI and Moblin, and a growing number of independent developers and community projects.

“The GNOME Mobile Initiative is at the heart of almost every important open source mobile effort going on in the industry today. The mainstream free software technologies such as GTK+, Gstreamer, matchbox and many other vibrant community-based projects are the linchpins of efforts like the ACCESS Linux Platform, Nokia’s Maemo platform, and the LiMo Foundation Platform. GNOME Mobile is the leading edge of development for the most exciting device space in decades, the rapidly growing world of open source-based mobile devices,” said David “Lefty” Schlesinger,
Director of Open Source Technologies for ACCESS Co., Ltd., and a member of the LiMo Foundation Architectural Council.