Matt Asay of Alfresco recently observed differences in the adoption of FOSS. He sees a lot of adoption in Europe by governments who support the ideals of FOSS whereas he claims that adoption in North America is mainly driven by enterprises "seeking to lower costs and maximize innovation". His comments remind me of a discussion I once had with someone a few years ago. We didn't talk so much about adoption, but rather about development of FOSS. Our conclusion was that the main driving forces behind FOSS are not the same in different parts of the world. Specifically, it seemed that FOSS development in North America was heavily driven by corporations, whereas FOSS development in Europe was mostly community driven. Finally, governments in Asia and other emerging markets, like Brazil, saw FOSS as a great opportunity to establish a healthy software development industry in their own countries and as a way to become more independent from the US.

Today, this picture may be different and clearly there are multiple driving forces in a region. The EU in particular has been pushing FOSS in recent times for public administration and they are funding a number of research projects related to FOSS, such as FLOSSMETRICS, SQO-OSS and QualiPSo (more on these later). This may explain why Matt Asay is seeing so much adoption of FOSS by governments in Europe. Nevertheless, in terms of development I'd still say that the community is the driving force in Europe. I find the differences between the FOSS community in Europe and in the US quite amazing. It seems to me that the FOSS community is much more tightly connected in Europe than in the US. In Europe, we have a lot of community events for FOSS (both for users and developers). FOSDEM is one of the largest FOSS events in Europe, but almost every country in Europe has their own event (or even several) related to Linux and FOSS. Many of them are organized by the community and entrance is often free of charge. Contrast this to the US, which has much fewer events and the most prominent one, OSCON, sets you back around $1000. I'm aware of OLS and SCALE, and have heard many good things about the LinuxFest Northwest and Ohio LinuxFest, but it's still not comparable to what you can find in Europe.

Some countries in Asia are very keen on participating in the FOSS community but I don't see many individual contributions from there (with the major exception of Japan, which has a very strong FOSS community). However, governments are providing incentives for companies to get involved. Unfortunately, there are many obstacles. For example, the open way in which most FOSS projects are performed may be in conflict to how some Asian cultures work. Also, FOSS development is often quite hostile (with heated discussions and arguments), which may be a problem to some. Finally, proficiency of English and access to broadband Internet connections are a problem in some Asian countries. Nevertheless, it's important to engage with developers in Asia and to help them contribute to and join our world-wide community.

In summary, I think there are important differences with regards to the adoption of FOSS and the way development communities work in different parts of the world. These differences will have an important impact on the governance of FOSS. Since FOSS is a world-wide phenomenon, we need to develop a better understanding of issues that various people face. I think that FOSSBazaar will be a great venue to hear from people all around the world and to learn what the obstacles are to wide adoption of FOSS in their countries.

(Originally published on FOSSBazaar)

Here are a number of links to various postings and other information related to my research about release management in free software projects:

• Bruce Byfield wrote an article about my research for Linux.com.
• I recently gave a presentation about my research at Google as part of their Tech Talks series. A video of this talk is now available.
• Matt Asay briefly discussed my work in his blog and relates it to commercial and proprietary software development. Roberto Galoppini relates my findings to his work and concludes that hierarchy is a must. Murray Cumming summarizes what he believes are the key findings of my research.

In previous times, most Linux distributions and other free software projects relied on the XFree86 system. Over the years, the structures within the XFree86 project became were rigid and the project failed to innovate and keep up with the pace of the wider free software community. When XFree86 changed its license in February 2004, the active community and the majority of vendors quickly moved to X.org. X.org is a very active community and they decided to break up the monolithic code base and to adopted a more modern build system. As of X.org 7.0, the project moved to the modular system in which components are developed and released separately. Effectively, the project introduced a development mechanism which features two release mechanisms: individual components can be released as needed and there is an overall release of X.org in which all stable components are put together. These roll-up releases take place every six months.

Past problems

• XFree86 made only infrequent releases every few years, had no plan, and the project's structures were very rigid.
• The code base was huge and monolithic. It had an archaic build system that few new developers were comfortable with. This made it hard for new contributors to get involved and was bad for testing.

Solutions

• X.org moved from a monolithic to a modular system. This made it easier to perform testing and it made it possible to give contributors write access to specific components.
• The move to the modular system allowed them the introduction of two release mechanisms: individual components can make releases on an ongoing basis and roll-up releases take place every six months.
• The roll-up releases have a fall back mechanism in case a specific component is not ready for release: the former release of this component can be incorporated.

Outstanding problems

• The project needs to work the interface between the server and drivers, so updates to hardware drivers can be released more often than other components.

Plone is a content management system that is built on the powerful Zope application server. The project experienced many delays with its 2.1 release. This made it difficult for Plone consultants to choose whether to use the old release or wait for the new one, and users faced many upgrade issues when 2.1 was finally released and had many changes. Partly to address these problems and partly in order to sync their releases with Zope, they decided in 2005 to move to a six month time based release cycle.

Past problems

• Releases, in particular version 2.1, took a long time to get out.
• Releases had many changes and caused some migration problems.
• Many Plone developers work as consultants building web sites. Because of the unpredictability of Plone, it was difficult for them to decide which version to use for future projects.

Solutions

• Plone moved to a time based release, partly because the Zope framework on which it builds has done so. This allows them to make use of new Zope features in their software.
• Time based releases allowed the project to implement more structure. For example, new features have to be proposed as a Plone Improvement Proposals (PLIP) which the framework team reviews.
• Deadlines have motivated developers to get their features done within a certain time frame.
• Plone consultants can decide in advance which version of Plone to use for future commercial projects.

Outstanding problems

• As the project has moved to time based releases only recently, they still need to show whether they can consistently release on time.

OpenOffice.org is an office suite offering various integrated applications, such as a word processor and a spreadsheet. Originally developed by StarDivision, StarOffice was acquired by Sun who released it as free software as OpenOffice.org in July 2000. While Sun still maintains fairly tight control over the development of OpenOffice.org, many other vendors, in particular Novell, are important contributors to the project. The project had a fairly long release cycle of about 18 months to accommodate StarOffice, the commercial product from Sun. There were many delays, making it hard for vendors to decide which version to include. OpenOffice.org moved to a three month release cycle after their long-delayed 2.0 release, published 26 months after 1.1. The new release cycle is viewed as a positive development by contributors who get their features and fixes out to users faster. Nevertheless, at the end of 2006 a discussion took place in which a six month interval was suggested. Apparently users didn't want new features every three months and the short interval between releases put a lot of pressure on the QA team.

Past problems

• The long release cycle of 18 months, bound to the commercial StarOffice product, meant that little testing occurred for a long time because developers believed the release was far away.
• Many changes accumulated during the long development phase, making testing towards the end very difficult, and leading to a `big bang' release.
• Features were put in very late, even during the beta cycle, because of the perceived 18 month delay to the next release.
• There was very little code review. The QA team only tested the program.
• Vendors shipped unreleased versions because the significant delays during the 2.0 cycle made planning impossible.

Solutions

• After the 2.0 release, the project moved to a 3 month release interval. This model promises a tight feedback loop with users.
• Because planning is now possible, collaboration between vendors on the same code base is far easier.
• The faster release cycle and more collaboration among vendors has promoted code review.
• Motivation in the project has increased because people see their contributions getting out to users within a reasonable time.
• The release process has become more transparent, allowing voluntary contributors to take a more active part in release preparations.

Outstanding problems

• There are discussions about changing the release interval to six months. There is some evidence that some users do not want new features every three months and that the aggressive release cycle of three months puts a lot of pressure on the QA team.

The Linux kernel project has seen major changes to its development and release strategy in the last few years, in particular since the first stable release of the 2.6 series in December 2003. This series was opened almost three years after the 2.4 series in January 2001, which many felt was too long. A problem that resulted from the long interval between major releases was that vendors had to back- and forward-port a lot of patches. Nowadays, major development happens on the 2.6 series and new releases are published every three to four months. This new development model has faced much controversy. While some people, in particular developers of the kernel, claim that the new model is working very well, some users are worried about the number of significant changes and lack of stability in the kernel. Andrew Morton, a lead developer, has expressed several times that he believes the kernel is getting buggier.

Past problems

• Because of the long release cycle, many changes accumulated. It was hard to get the development stable and there were few testers.
• Features got out very slowly because of the long release cycle.
• Hardware support and crucial features had to be backported to the latest stable kernel.
• Vendors backported many features to their own releases. The code base from different vendors diverged a lot from each other and from the official development version.

Solutions

• New versions are now released every two or three months. There is a two week merge window after each release during which new features are accepted; subsequently, the focus is on stabilization.
• There is now a steady flow of code into production and many people get to test the new code.
• Features get out more quickly.
• Vendors can directly work with current releases and get their changes into official versions easily.

Outstanding problems

• There is no long-term stable version based on the 2.6 kernel. This is being addressed with Adrian Bunk's 2.6.16 long-term maintenance tree but whether it will have a big impact remains to be seen.
• Regressions between versions are introduced more frequently. Better control and tracking of regressions are needed.
• The Bugzilla bug tracker needs to be integrated better into the development process and it would be helpful to have a QA person. In January 2007, Google announced that they're looking for a kernel QA person who'd work together with Andrew Morton.

GNOME provides a complete desktop environment that is easy to use. The project experienced major problems, such as delays, during its 1.x series and in particular during the preparation of its 2.0 release. GNOME introduced time based releases after its 2.0 release and has significantly improved release management. The project has published time based releases every six months for a number of years now and is considered as the reference model for a good implementation of time based release management. They have shown that it's possible to set and meet deadlines in volunteer projects and to release on time.

Past problems

• The main goal of version 2.0 was to change internal interfaces but after more than a year of work the project felt they had to deliver more user-visible changes, therefore leading to delays.
• Developers were disappointed with delays and that their work was not available to users.
• It was not clear what was going on. Developers constantly had to ask about the release status.
• Freezes were announced and people worked towards them but then they were delayed. Freezes also often came unexpectedly.
• Vendors had deadlines but the GNOME schedule was unpredictable. Vendors did not know whether they should aim for the next version or focus on the previous version and backport fixes.
• Vendors had to backport many changes to a version that the GNOME project considered as old.

Solutions

• The project introduced a rigorous schedule promising a release every six months.
• There was a core team so few people had to agree to the introduction of a schedule.
• GNOME introduced policies to keep the development tree fairly stable.
• The project introduced the idea of reverting: if a feature was not ready on a certain cut-off date, it would be taken out again.
• The project gained credibility because releases were actually performed on time.
• The schedule allowed vendors better planning, and hence vendors could implement their features on the main development tree.

Outstanding problems

• The six month schedule has been successful in the delivery of incremental updates. There are some concerns whether this release cycle makes the project less innovative and ambitious regarding major changes that would lead to GNOME 3.0.

The GNU Compiler Collection (GCC) is a compiler suite which supports a number of programming languages, such as C and C++. It is a very important development tool and is the standard compiler among free software projects. GCC was quite stagnant in the middle of the 1990s until the EGCS project formed. EGCS took over development of the official GCC in October 1998 and instituted rigorous processes, such as high levels of peer review, and created a steering committee which has the power to appoint maintainers and make important decisions. In theory, the project follows a time based release with an interval of six months. In practice, the project has released only one new version every year in recent times.

Past problems

• The GCC project suffered from a closed development style in the past: few people could make code changes and the mailing list was by invitation only.
• There was a long time between releases and development snapshots, which contained bug fixes and features, were not available to the public.
• When development opened and picked up, significant code changes were often made which required a long stabilization phase.

Solutions

• The project moved to a more open development style and established a steering committee which has the power to appoint new maintainers and make important decisions.
• The development process was divided into three stages in order to coordinate code submissions and keep the development tree reasonable stable. Major changes that developers wish to make during stage 1 and stage 2 have to be proposed on the project's wiki. Proposals are reviewed by the release manager who also assign a sequence to proposed projects which says when they can be applied to the development tree.
• All patches are peer reviewed on the development mailing list and need approval before they can be applied. The steering committee appoints maintainers who can approve patches that touch particular areas. There are maintainers for the C front-end, various port maintainers, and maintainers for other areas.
• Each development stage lasts two months, theoretically yielding a release every six months. A regular release cycle ensures that it is not the end of the world if a feature does not make it into the following release.

Outstanding problems

• The release manager is busy and has not pushed the release forwards as much as would be possible.
• The branch criteria may need revision to make it easier to create a branch which leads to the next stable version. At the moment, the branch criteria is to have 100 or less regressions on the development tree. However, this number may include regressions that are also present in previous releases, which makes it fairly hard to reach this criteria.

The aim of Debian is to integrate software produced by other projects to create a complete operating system based on free software. In recent years, the project has faced increasingly delayed and unpredictable releases. Most notably, the release process of Debian 3.1 was characterized by major delays. Initially announced for December 1, 2003, the software was finally released in June 2005: a delay of one and a half years. Since then, the project has made a number of improvements to its release process.

Past Problems

• Release management was not very organized and release updates were posted only infrequently. Because of this and the lack of a roadmap, freezes were often announced out of the blue.
• Due to the unorganized nature of the release, several new and unexpected blockers were found during the release process, leading to delays.
• The unexpected delays meant that software was frozen for a long time, in the case of Debian 3.1 for over a year. When this release was finally published, many components were already out of date and would often not meet user demands.
• The fact that Debian has experienced significant delays with several of its recent releases has led to problems with the image of the project. There is a perception that Debian is slow and cannot meet deadlines. This is also associated with frustration in the developer and user community.

Solutions

• The project has implemented better release management structures. In particular, Debian moved from a single release manager to a team during the 3.1 release cycle. The release is now handled by two release managers with the help of several release assistants.
• A release date for the next release was set well in advance and there is more planning.
• Release announcements are sent more frequently and various information sources have been implemented through which developers can stay informed about the release status.
• Release targets have been defined better and there is a distinction between blockers and goals. Only blockers hold up the release whereas goals can be postponed for a future release. For example, the XFree86 to X.org transition and the integration of AMD64 support were blockers, while LSB 3.1 compatibility and SE Linux support were only goals for Debian 4.0.
• The release team has shifted the responsibility of achieving targets to specific developers and teams. There are also better criteria now defining these responsibilities. For example, criteria have been published that architectures need to fulfil for inclusion in Debian 4.0 and there is a page indicating the status of each architecture.
• The release team is increasingly giving encouragement to developers to help out with software packages and bugs which normally do not fall into their domain.
• A staged freeze has been implemented according to which software is frozen in stages according to its importance. The toolchain and base system are frozen earlier than the majority of package. The majority of software is now frozen when most blockers have been resolved.
• The use of the experimental repository has been promoted to make sure that the main development repository is in a good shape most of the time.

Outstanding problems

• Developers still need to be convinced that targets can be met, that deadlines are real and that Debian can release on time.

As I'm nearing the completion of my PhD, I'm starting to write about the findings from my research. The aim of my PhD is to investigate quality improvement in free software projects. After some exploratory studies about quality in free software, I decided to focus on release management as one problematic area in which improvement is possible. My approach is to study the development process of projects and to investigate ways how these processes can be improved. The hope is that such improvements will lead to increased levels of quality in the software produced by free software projects.

With regard to release management, I'm focusing especially on large and complex projects, in which hundreds of volunteers need to be coordinated during release preparations. I've been looking particularly at time based release management since my exploratory studies have shown that there is significant interest in this release management strategy. The main question of my research is how and why time based release management works for large free software projects.

In my first series of postings, I will introduce the seven free software projects I have studied in depth during my research. For each project, I will briefly summarize what problems the project faced in the past, what changes they have implemented to address these problems, and what outstanding problems they still have.

Projects: Debian | GCC | GNOME | Linux kernel | OpenOffice.org | Plone | X.org