JaikusI’m currently working on a report comparing the Linux Kernel, KDE, Dokuwiki and Ruby on Rails in terms of how they work and perform as open source projects. We have collected data from the mailing lists, issue tracking systems and source control systems of each project (October 2006 - September 2007), and while I don’t have anything groundbreaking to report, I do have some food for thought for the Ruby on Rails community.
These four pies clearly show that Linux - not surprisingly - is the largest projects in terms of changesets (patches committed to the code base), messages (emails sent to the core mailing list) and individual senders (different people who has sent messages to the mailing list). But in that same year, almost twice as many issues (which is our neutral term for what is called tickets in Trac) has been opened in the Ruby on Rails Trac system as in the Linux Kernel Bugzilla system, despite the fact more than 10 ten time as many changes has been accepted into the Linux code base.
“Wait a minute!” you’ll probably object if you know a bit about how things roll in Linux-land, “they use mailing lists for that stuff on the Linux Kernel, so issues are only used by a few to report bugs.” And that’s absolutely true. The activity level of the Linux Kernel mailing list is more than 30 times the one on the Rails Core mailing list, and one out four messages either contain a patch or is directly related to one.
So okay, the Linux-community handles changes in one way, and the Rails-community in another - big deal, it seems to work out for both communities. Now, web-based issue tracking systems like Trac or Bugzilla are supposed to be the “new” way to manage open source projects, right? Linux is just stuck in the old mailing list workflow, but surely that must Linux much less efficient than Rails, because there is no overview of open issues and patches, no reports and no statistics? Well, not really.
The bar chart above shows the mean time to change for each of the projects. It is calculated as the average time span elapsing from an issue is opened in the issue tracking system till it is closed again. On average, it takes a patch 74 days to get into Linux code base and distributed as part of the worlds most popular operating system for servers. For Ruby on Rails, it “only” takes 54 days, so clearly the Rails-communities way of doing thing is more efficient. Or is it?
It is almost twice as fast to get a patch into the Dokuwiki code base as into the one of Rails. “Well, that’s a fairly small project with the creator also being the main contributor, so that’s not so strange” you might say in an attempt to defend your favourite open source community. Okay, but how about KDE? You can get a patch into the KDE core more than three times as fast. Now, to be fair, this is not the entire KDE suite of applications, but only the core of KDE, so most of the contributors has direct commit access themselves. But KDE and Dokuwiki aside, shouldn’t it still be faster than 54 days on average to a patch into the Ruby on Rails framework?
“Okay, so the average is 54 days, but that’s just because a few issues are left there like forever distorting the average” you might say. Well, first of all only issues that has been opened within the year and closed with a resolution if fixed is included the metric. Second, as the histogram above shows, around half of the issues are fixed within 12 days (the cumulative percentage is almost 50 at the first high bar, and that’s 12 days, not 24 as Excel insist on putting below the bar), but the other half takes close to forever to get closed.
So food for thought I said. I’m not pointing fingers at the Ruby on Rails community, of which I consider myself to be a part, only saying that we might want to think about if using Trac for everything and only having 12 people with commit rights is the best approach? Think about it, and if you have comments, critique, ideas, please do post them here so that we might use them in the final report.
PS: The period of time ends a short while after the whole Report #12 procedure was announced, so hey, that might fix everything …
I think the slow development process is due in part to the large number of people submitting poorly written patches or no patch at all, leaving the Rails community to figure out the problem for them.
Using their new procedure for submitting patches might help to speed things up. One of my patches was accepted just 2 hours after I submitted it.
I think they could also significantly speed things up if they’d migrate to a distributed version control system (whether it’s git, mercurial, bzr, etc). Almost anything is better than subversion.
Comment by Andrew Bennett — November 26, 2007 @ 3:26 am
It would be interesting if you could run the numbers for Rails again once Report 12 have been in place for some while.
Report 12 in itself is a sign that the lengthy process have been recognized by the Core team and they’re attempting to address it. Running the numbers again could indicate wether Report 12 is successful.
Comment by Jakob S — November 26, 2007 @ 3:17 pm