Joining HP

I joined HP's Open Source and Linux Organization (OSLO) at the beginning of the month as an Open Source Community Expert. I spent the last three weeks in Ft. Collins for some training and to get to know the team, but normally I'll be working from home in Europe. The main focus of my work will be governance issues related to the deployment of FOSS in enterprise environments. However, I've already started to get involved in a number of other areas I'm interested in, such as community relations. I'm pretty excited about my new position because it is a good fit for my skills and interests.

GCC 4.3 related build problems: duplicate function parameters

GCC 4.3 now prints an error message when C++ code contains duplicate function parameter names in function prototypes. You'd think that's a pretty obvious problem, but we still have about 15 packages in the archive that contain such code.

Anyway, for code like this:

class foo {
    foo(int w, int w);
};

GCC will as of 4.3 show the following error message:

t.cc:2: error: multiple parameters named 'w'

N2100 network troubles due to r8169 bug

Daniel Smolik reported problems with NFS on a Thecus N2100 a few days ago. I confirmed the bug with 2.6.18 and 2.6.22 but found that the problem didn't occur with 2.6.23. I started a git bisect and a few hours later located a fix that Francois Romieu, the maintainer of the r8169 driver, had put in. Apparently the driver had some confusion regarding hardware and IP header alignment. The good news is that the patch is very small and can easily be backported to our 2.6.18 kernel in stable. Also, packages of 2.6.23 were uploaded today and should be in the archive soon.

Tips on reducing memory and running Linux on a flash device

David Härdeman has contributed two really valuable guides to my Debian on NSLU2 site that will be of great interest to many NSLU2 users:

• The first guide is for people running Linux on a flash device, such as a USB flash stick. Since flash only supports a limited number of writes, it is important to reduce the wear and tear on the underlying flash device. David's guide gives some practical advice of how to achieve this.
• The second guide gives a few tips and tricks that can be used to reduce the amount of RAM a system uses. This is particularly interesting to NSLU2 users since 32 MB of RAM is not a lot.

These guides mention Debian but most of the advice applies to other NSLU2 firmware flavours and to Linux in general.

NSLU2 tar ball for Debian 4.0r1

I updated the tar ball for the manual Debian/NSLU2 installation to Debian 4.0r1 (including the new 2.6.18-5 kernel) today. The proprietary NPE microcode from Intel is now included in the tar ball since the code no longer has a click-through license.

Dr Michlmayr

Having passed all requirements for the Doctor of Philosophy, I had my graduation today.

GCC 4.3 related build problems: pedantic preprocessor warnings in C++

Update: This change has been reverted from 4.3 again and won't be in the final release of 4.3.

GCC 4.3 changed pedantic preprocessor warnings in C++ code into errors, in line with the C++ front-end. Because of this change, preprocessor warnings are now errors.

This change has caused at least the twofollowing errors which are all fairly common:

error: "xxx" redefined

It's not allowed to #define something twice, as in this example:

#define TEST "foo"
#define TEST "bar"

This code will lead to the following error:

test2.cc:2:1: error: "TEST" redefined
test2.cc:1:1: error: this is the location of the previous definition

Most typically, this occurs because something is defined in a header that is included and then redefined in the current source file. It can also happen when a #define is put in the code but also on the command line, e.g. via -DTEST="bar". This will lead to:

test2.cc:1:1: error: "TEST" redefined
<command line>:1:1: error: this is the location of the previous definition

The correct solution is to avoid redefinitions directly or to use #ifndef to make sure that something has not been defined already.

extra tokens at end of #endif directive

While an #ifdef obviously needs a second parameter, it's not allowed to give one to #else and #endif. Unfortunately, this is commonly done as a reminder for what is being tested for:

#ifdef TEST
#endif TEST

This code will lead to the following error:

test3.cc:2:8: error: extra tokens at end of #endif directive

GCC 4.3 related build problems: missing #include

In GCC 4.3 the C++ header dependencies have been cleaned up. In the past, compilation would often be quite slow because including a simple header would indirectly include a lot of other headers, even if they were not needed to compile the current code. Many headers have now been cleaned up with the result that compilation is quicker. The downside is that programmers cannot rely on indirect inclusion of headers they may need anymore. Instead, everything that is needed has to be directly referenced with an #include.

Typical errors look like these:

error: 'find' is not a member of 'std'
error: 'exit' was not declared in this scope

Below is a table showing which header needs to be included for a number of common functions:

Functions and defines	Header
find, for_each, sort	algorithm
isalnum, toupper	cctype
INT_MIN, RAND_MAX	climits
printf	cstdio
atoi, free, rand	cstdlib
EXIT_FAILURE	cstdlib
strcmp, memcpy	cstring
auto_ptr	memory
fd_set, mode_t	sys/types.h
typeid	typeinfo

GCC 4.3: common build failures

Even though GCC 4.2 has seen many delays and still hasn't been released (although it's finally getting close), a lot of development has already been done on trunk for the release of GCC 4.3. I started testing 4.3 in March and in addition to finding and reporting compiler issues I have filed bugs on packages that fail to build with GCC 4.3. The idea is to give people an advance warning and to be ready for GCC 4.3 when it will be released.

We have almost 500 bugs related to GCC 4.3 already, mostly due to a clean-up of C++ headers in GCC that has exposed sloppy programming in many programs. Fortunately, these bugs are easy to fix. As I'm finding common mistakes in packages, I will document these problems and way to make sure a program is ready for GCC 4.3.

Common problems

• Missing #include directives
• Duplicate parameters in function prototypes

Update: there is now a GCC 4.3 porting guide that describes the most common problems people will face when moving to GCC 4.3.

Links

Here are a number of links to various postings and other information related to my research about release management in free software projects:

• Bruce Byfield wrote an article about my research for Linux.com.
• I recently gave a presentation about my research at Google as part of their Tech Talks series. A video of this talk is now available.
• MORITA Hajime kindly translated my blog postings about my research to Japanese.
• Matt Asay briefly discussed my work in his blog and relates it to commercial and proprietary software development. Roberto Galoppini relates my findings to his work and concludes that hierarchy is a must. Murray Cumming summarizes what he believes are the key findings of my research.

X.org

In previous times, most Linux distributions and other free software projects relied on the XFree86 system. Over the years, the structures within the XFree86 project became were rigid and the project failed to innovate and keep up with the pace of the wider free software community. When XFree86 changed its license in February 2004, the active community and the majority of vendors quickly moved to X.org. X.org is a very active community and they decided to break up the monolithic code base and to adopt a more modern build system. As of X.org 7.0, the project moved to the modular system in which components are developed and released separately. Effectively, the project introduced a development mechanism which features two release mechanisms: individual components can be released as needed and there is an overall release of X.org in which all stable components are put together. These roll-up releases take place every six months.

Version	Date	Months
7.0	2005-12-21
7.1	2006-05-22	5
7.2	2007-02-15	9

Past problems

• XFree86 made only infrequent releases every few years, had no plan, and the project's structures were very rigid.
• The code base was huge and monolithic. It had an archaic build system that few new developers were comfortable with. This made it hard for new contributors to get involved and was bad for testing.

Solutions

• X.org moved from a monolithic to a modular system. This made it easier to perform testing and it made it possible to give contributors write access to specific components.
• The move to the modular system allowed them the introduction of two release mechanisms: individual components can make releases on an ongoing basis and roll-up releases take place every six months.
• The roll-up releases have a fall back mechanism in case a specific component is not ready for release: the former release of this component can be incorporated.

Outstanding problems

• The project needs to work the interface between the server and drivers, so updates to hardware drivers can be released more often than other components.

Plone

Plone is a content management system that is built on the powerful Zope application server. The project experienced many delays with its 2.1 release. This made it difficult for Plone consultants to choose whether to use the old release or wait for the new one, and users faced many upgrade issues when 2.1 was finally released and had many changes. Partly to address these problems and partly in order to sync their releases with Zope, they decided in 2005 to move to a six month time based release cycle.

Version	Date	Months
1.0	2003-02-06
2.0	2004-03-23	13
2.1	2005-09-06	17
2.5	2006-06-16	9

Past problems

• Releases, in particular version 2.1, took a long time to get out.
• Releases had many changes and caused some migration problems.
• Many Plone developers work as consultants building web sites. Because of the unpredictability of Plone, it was difficult for them to decide which version to use for future projects.

Solutions

• Plone moved to a time based release, partly because the Zope framework on which it builds has done so. This allows them to make use of new Zope features in their software.
• Time based releases allowed the project to implement more structure. For example, new features have to be proposed as a Plone Improvement Proposals (PLIP) which the framework team reviews.
• Deadlines have motivated developers to get their features done within a certain time frame.
• Plone consultants can decide in advance which version of Plone to use for future commercial projects.

Outstanding problems

• As the project has moved to time based releases only recently, they still need to show whether they can consistently release on time.

OpenOffice.org

OpenOffice.org is an office suite offering various integrated applications, such as a word processor and a spreadsheet. Originally developed by StarDivision, StarOffice was acquired by Sun who released it as free software as OpenOffice.org in July 2000. While Sun still maintains fairly tight control over the development of OpenOffice.org, many other vendors, in particular Novell, are important contributors to the project. The project had a fairly long release cycle of about 18 months to accommodate StarOffice, the commercial product from Sun. There were many delays, making it hard for vendors to decide which version to include. OpenOffice.org moved to a three month release cycle after their long-delayed 2.0 release, published 26 months after 1.1. The new release cycle is viewed as a positive development by contributors who get their features and fixes out to users faster. Nevertheless, at the end of 2006 a discussion took place in which a six month interval was suggested. Apparently users didn't want new features every three months and the short interval between releases put a lot of pressure on the QA team.

Version	Date	Months
1.0	2002-05-01
1.1	2003-09-02	16
2.0	2005-10-20	26
2.0.1	2005-12-21	2
2.0.2	2006-03-08	3
2.0.3	2006-06-29	4
2.0.4	2006-10-13	3
2.1.0	2006-12-12	2

Past problems

• The long release cycle of 18 months, bound to the commercial StarOffice product, meant that little testing occurred for a long time because developers believed the release was far away.
• Many changes accumulated during the long development phase, making testing towards the end very difficult, and leading to a `big bang' release.
• Features were put in very late, even during the beta cycle, because of the perceived 18 month delay to the next release.
• There was very little code review. The QA team only tested the program.
• Vendors shipped unreleased versions because the significant delays during the 2.0 cycle made planning impossible.

Solutions

• After the 2.0 release, the project moved to a 3 month release interval. This model promises a tight feedback loop with users.
• Because planning is now possible, collaboration between vendors on the same code base is far easier.
• The faster release cycle and more collaboration among vendors has promoted code review.
• Motivation in the project has increased because people see their contributions getting out to users within a reasonable time.
• The release process has become more transparent, allowing voluntary contributors to take a more active part in release preparations.

Outstanding problems

• There are discussions about changing the release interval to six months. There is some evidence that some users do not want new features every three months and that the aggressive release cycle of three months puts a lot of pressure on the QA team.

Linux kernel

The Linux kernel project has seen major changes to its development and release strategy in the last few years, in particular since the first stable release of the 2.6 series in December 2003. This series was opened almost three years after the 2.4 series in January 2001, which many felt was too long. A problem that resulted from the long interval between major releases was that vendors had to back- and forward-port a lot of patches. Nowadays, major development happens on the 2.6 series and new releases are published every three to four months. This new development model has faced much controversy. While some people, in particular developers of the kernel, claim that the new model is working very well, some users are worried about the number of significant changes and lack of stability in the kernel. Andrew Morton, a lead developer, has expressed several times that he believes the kernel is getting buggier.

Version	Date	Months
1.0	1994-03-14
1.2	1995-03-07	12
2.0	1996-06-09	15
2.2	1999-01-25	31
2.4	2001-01-04	23
2.6	2003-12-17	35

Past problems

• Because of the long release cycle, many changes accumulated. It was hard to get the development stable and there were few testers.
• Features got out very slowly because of the long release cycle.
• Hardware support and crucial features had to be backported to the latest stable kernel.
• Vendors backported many features to their own releases. The code base from different vendors diverged a lot from each other and from the official development version.

Solutions

• New versions are now released every two or three months. There is a two week merge window after each release during which new features are accepted; subsequently, the focus is on stabilization.
• There is now a steady flow of code into production and many people get to test the new code.
• Features get out more quickly.
• Vendors can directly work with current releases and get their changes into official versions easily.

Outstanding problems

• There is no long-term stable version based on the 2.6 kernel. This is being addressed with Adrian Bunk's 2.6.16 long-term maintenance tree but whether it will have a big impact remains to be seen.
• Regressions between versions are introduced more frequently. Better control and tracking of regressions are needed.
• The Bugzilla bug tracker needs to be integrated better into the development process and it would be helpful to have a QA person. In January 2007, Google announced that they're looking for a kernel QA person who'd work together with Andrew Morton.

GNOME

GNOME provides a complete desktop environment that is easy to use. The project experienced major problems, such as delays, during its 1.x series and in particular during the preparation of its 2.0 release. GNOME introduced time based releases after its 2.0 release and has significantly improved release management. The project has published time based releases every six months for a number of years now and is considered as the reference model for a good implementation of time based release management. They have shown that it's possible to set and meet deadlines in volunteer projects and to release on time.

Version	Date	Months
1.0	1999-03-03
1.2	2000-05-25	15
1.4	2001-04-02	10
2.0	2002-06-27	15
2.2	2003-02-06	7
2.4	2003-09-11	7
2.6	2004-03-31	7
2.8	2004-09-15	6
2.10	2005-03-09	6
2.12	2005-09-07	6
2.14	2006-03-15	6
2.16	2006-09-06	6
2.18	2007-03-14	6

Past problems

• The main goal of version 2.0 was to change internal interfaces but after more than a year of work the project felt they had to deliver more user-visible changes, therefore leading to delays.
• Developers were disappointed with delays and that their work was not available to users.
• It was not clear what was going on. Developers constantly had to ask about the release status.
• Freezes were announced and people worked towards them but then they were delayed. Freezes also often came unexpectedly.
• Vendors had deadlines but the GNOME schedule was unpredictable. Vendors did not know whether they should aim for the next version or focus on the previous version and backport fixes.
• Vendors had to backport many changes to a version that the GNOME project considered as old.

Solutions

• The project introduced a rigorous schedule promising a release every six months.
• There was a core team so few people had to agree to the introduction of a schedule.
• GNOME introduced policies to keep the development tree fairly stable.
• The project introduced the idea of reverting: if a feature was not ready on a certain cut-off date, it would be taken out again.
• The project gained credibility because releases were actually performed on time.
• The schedule allowed vendors better planning, and hence vendors could implement their features on the main development tree.

Outstanding problems

• The six month schedule has been successful in the delivery of incremental updates. There are some concerns whether this release cycle makes the project less innovative and ambitious regarding major changes that would lead to GNOME 3.0.

GCC

The GNU Compiler Collection (GCC) is a compiler suite which supports a number of programming languages, such as C and C++. It is a very important development tool and is the standard compiler among free software projects. GCC was quite stagnant in the middle of the 1990s until the EGCS project formed. EGCS took over development of the official GCC in October 1998 and instituted rigorous processes, such as high levels of peer review, and created a steering committee which has the power to appoint maintainers and make important decisions. In theory, the project follows a time based release with an interval of six months. In practice, the project has released only one new version every year in recent times.

Version	Date	Months
3.0	2001-06-18
3.1	2002-05-15	11
3.2	2002-08-14	3
3.3	2003-05-13	9
3.4.0	2004-04-18	11
4.0.0	2005-04-20	12
4.1.0	2006-02-28	10

Past problems

• The GCC project suffered from a closed development style in the past: few people could make code changes and the mailing list was by invitation only.
• There was a long time between releases and development snapshots, which contained bug fixes and features, were not available to the public.
• When development opened and picked up, significant code changes were often made which required a long stabilization phase.

Solutions

• The project moved to a more open development style and established a steering committee which has the power to appoint new maintainers and make important decisions.
• The development process was divided into three stages in order to coordinate code submissions and keep the development tree reasonable stable. Major changes that developers wish to make during stage 1 and stage 2 have to be proposed on the project's wiki. Proposals are reviewed by the release manager who also assign a sequence to proposed projects which says when they can be applied to the development tree.
• All patches are peer reviewed on the development mailing list and need approval before they can be applied. The steering committee appoints maintainers who can approve patches that touch particular areas. There are maintainers for the C front-end, various port maintainers, and maintainers for other areas.
• Each development stage lasts two months, theoretically yielding a release every six months. A regular release cycle ensures that it is not the end of the world if a feature does not make it into the following release.

Outstanding problems

• The release manager is busy and has not pushed the release forwards as much as would be possible.
• The branch criteria may need revision to make it easier to create a branch which leads to the next stable version. At the moment, the branch criteria is to have 100 or less regressions on the development tree. However, this number may include regressions that are also present in previous releases, which makes it fairly hard to reach this criteria.

Debian

The aim of Debian is to integrate software produced by other projects to create a complete operating system based on free software. In recent years, the project has faced increasingly delayed and unpredictable releases. Most notably, the release process of Debian 3.1 was characterized by major delays. Initially announced for December 1, 2003, the software was finally released in June 2005: a delay of one and a half years. Since then, the project has made a number of improvements to its release process.

Version	Date	Months
1.1	1996-06-17
1.2	1996-12-12	6
1.3	1997-06-02	6
2.0	1998-07-24	14
2.1	1999-03-09	7
2.2	2000-08-14	17
3.0	2002-07-19	23
3.1	2005-06-06	35

Past Problems

• Release management was not very organized and release updates were posted only infrequently. Because of this and the lack of a roadmap, freezes were often announced out of the blue.
• Due to the unorganized nature of the release, several new and unexpected blockers were found during the release process, leading to delays.
• The unexpected delays meant that software was frozen for a long time, in the case of Debian 3.1 for over a year. When this release was finally published, many components were already out of date and would often not meet user demands.
• The fact that Debian has experienced significant delays with several of its recent releases has led to problems with the image of the project. There is a perception that Debian is slow and cannot meet deadlines. This is also associated with frustration in the developer and user community.

Solutions

• The project has implemented better release management structures. In particular, Debian moved from a single release manager to a team during the 3.1 release cycle. The release is now handled by two release managers with the help of several release assistants.
• A release date for the next release was set well in advance and there is more planning.
• Release announcements are sent more frequently and various information sources have been implemented through which developers can stay informed about the release status.
• Release targets have been defined better and there is a distinction between blockers and goals. Only blockers hold up the release whereas goals can be postponed for a future release. For example, the XFree86 to X.org transition and the integration of AMD64 support were blockers, while LSB 3.1 compatibility and SE Linux support were only goals for Debian 4.0.
• The release team has shifted the responsibility of achieving targets to specific developers and teams. There are also better criteria now defining these responsibilities. For example, criteria have been published that architectures need to fulfil for inclusion in Debian 4.0 and there is a page indicating the status of each architecture.
• The release team is increasingly giving encouragement to developers to help out with software packages and bugs which normally do not fall into their domain.
• A staged freeze has been implemented according to which software is frozen in stages according to its importance. The toolchain and base system are frozen earlier than the majority of package. The majority of software is now frozen when most blockers have been resolved.
• The use of the experimental repository has been promoted to make sure that the main development repository is in a good shape most of the time.

Outstanding problems

• Developers still need to be convinced that targets can be met, that deadlines are real and that Debian can release on time.

Quality Improvement in Free Software: Release Management

As I'm nearing the completion of my PhD, I'm starting to write about the findings from my research. The aim of my PhD is to investigate quality improvement in free software projects. After some exploratory studies about quality in free software, I decided to focus on release management as one problematic area in which improvement is possible. My approach is to study the development process of projects and to investigate ways how these processes can be improved. The hope is that such improvements will lead to increased levels of quality in the software produced by free software projects.

With regard to release management, I'm focusing especially on large and complex projects, in which hundreds of volunteers need to be coordinated during release preparations. I've been looking particularly at time based release management since my exploratory studies have shown that there is significant interest in this release management strategy. The main question of my research is how and why time based release management works for large free software projects.

In my first series of postings, I will introduce the seven free software projects I have studied in depth during my research. For each project, I will briefly summarize what problems the project faced in the past, what changes they have implemented to address these problems, and what outstanding problems they still have.

Projects: Debian | GCC | GNOME | Linux kernel | OpenOffice.org | Plone | X.org

debian-installer rc2 for NSLU2 available

Frans Pop uploaded debian-installer a few days ago in preparation for RC2. RC2 will be released in a few days after some more testing and the creation of CD images. Since the NSLU2 doesn't need CD images and I've tested the installer already, I've made NSLU2 images of RC2 available.

Special thanks go to Gordon Farquharson who has done tremendous amounts of testing in the last few weeks and months. Since RC1, the following changes have been implemented:

• The kernel now uses the Ethernet driver developed by Christian Hohnstaedt. This driver is aimed for inclusion in the mainline kernel, even though it still needs quite a bit of work to get there.
• The kernel has been upgraded to 2.6.18. This version adds LED support for the NSLU2 and fixes real time clock support. The NSLU2 is pretty much fully supported with this kernel.
• The installer now has the ability to set the time zone during installation.
• Joey Hess has substantially updated the nslu2-utils package. Among other things, it has proper support for the LEDs now. I have added LABEL and UUID support for /etc/fstab and made the parsing code more robust, making it unlikely that a kernel upgrade will make your device unbootable.

Installation instructions are available.

Lemote's Fulong Miniature Computer

Yesterday I received a Fulong Miniature Computer from China. This device is produced by Lemote and is based on the Loongson chip, a MIPS-like CPU developed in China and previously known as the Godson. The Fulong mini-PC has a Loongson 2E CPU running at 666 Mhz, 256 MB RAM and a 2.5" 40 GB hard drive. There are the usual I/O ports, such as USB, PS/2, Ethernet, audio, VGA and TV out. The Fulong mini-PC ships with a customized version of Debian.

I'm really excited about the Loongson because it promises to make MIPS available on the desktop. In fact, the folks at Lemote already have a laptop based on the same motherboard used in the Fulong mini-PC. Lemote is also trying to establish a community around the Loongson. They have given 400 devices to partners and 600 to Loongson fans and developers in China. It seems that the community is quite active, but it's hard to tell because I don't read Chinese.

For now, I have taken some pictures of the Fulong, but in the coming months I look forward to integrating Loongson support into Debian.