Community Matters

Most every year, as we prepare for the kernel summit, this topic comes up. How do we ensure Linux doesn’t turn into the old boys club. How do we attract new developers and get them to grow into bigger roles into the developer community.

It’s a typical application of the ten thousand hour rule. You start working on the kernel. It will take you about ten thousand hours to become an expert and be truly able to work on the next level, owning a subsystem, be truly a leader. According to research by Anders Ericsson that is a fairly consistent threshold how long it takes to reach true greatness in any art form. Music, painting, computer programming.

If you manage to spend 20 hours a week hacking the kernel that will take you about ten years. At which point you will no longer be perceived as “new blood”. If you are one if the few people willing and able to hack 80 hours a week you can get there in about two and a half years and be one of the very few brilliant newcomers we see. Maybe one or two every other year.

So the next time people ask about the new blood, I think we should turn around and ask them if they are looking at this the right way.

This isn’t a new idea. Joe Wilcox has discussed this in his BetaNews article a couple of months ago.

But spending a lot of time at OSCON last week made it clear to me how true this observation has become – with all its implications.

The developers of Android and MeeGo are quite actively playing “Internet” to Apple’s “AOL”. Instead of inviting innovation and content onto their platform, Apple is focused on controlling every aspect. All under the guise of delivering a better user experience. At the same time throttling innovation and freedom for their users. And just as with AOL, at first that seemed like a good idea. You know, a well maintained garden, everything is pretty, none of that pesky dangerous (or seedy) stuff that is out there in the unregulated Internet.

But it turns out that people want that. Whether it is free access to applications (thanks to the Library of Congress, there is some good news for Apple users – but if you have to “jailbreak” your device in order to get the software you want, maybe you are using the wrong device to begin with). Or whether it is the ability to extend functionality (tethering, anyone?).

A year ago everyone was looking at Android and was writing them off. Look at all the apps that the iPhone has. Look at all that mind share. Today, Android is activating about 160k devices every day, the app store is growing like crazy and the traction in the eco system (and the non-stop comparisons) are showing that the tides have turned.

How will Apple prove that they’ve lost touch? Here are my predictions. They will continue to show their contempt for their customers. New feature in Mac OS 10.7 (rebranded as iOS7 or something): applications can only be installed from the Apple AppStore. Adding an additional monitor to your Mac will require an app (that of course you pay for). The iPad will only connect to Apple approved wireless networks. They will continue to prohibit network sharing between devices. It’s just like dial-up.

And what will the Linux players do to counter this? They will encourage innovation and new ideas. They will allow people to hack the devices that they bought, to make them better, weirder, different.

Yes, it took a while for the Internet to drive AOL into irrelevance. And similarly, it will take time for the mass of the customers to realize just how Apple is taking advantage of them. And there will continue to be some fanboyz. Hey, I just this week got email from someone with an AOL email address (and the “real name” in the email headers was their AOL email address again, this time in all caps)…

This has been an ongoing frustration for me for a long time.

I used to be a mutt user – still one of the best email clients out there. That is, if your email comes from an IMAP server and is mostly text-only. No images, HTML, links, etc. Yes, it can sort of kinda work with those things, but please, let’s get real.

Emacs and its various mail modes are of course an interesting option (especially as you can, in fact, display most everything inside modern emacs). And Notmuch is making handling tons of email even easier with decent emacs integration (yes, it’s very early in its development, but for things like reading lkml it is amazing).

Or you can go with Thunderbird (version 3 is really impressive, the tabbed UI takes a little getting used to but then worked rather well for me). Or claws-mail (fast but rather unstable and its single-threadedness really got me to hate it). Or even (yikes) Evolution. Sadly without a strong leader anymore and rather aimless for the last year or so.

But the problem is this – if you want to access work email as well as your personal stuff, chances are that you are forced to integrate with MS Exchange. I can give you tons of reasons why Exchange is a Really Bad Idea™, but of course your corporate IT department is likely to ignore those and tell you “Exchange it is”. And among the biggest flaws of Exchange is its rotten IMAP support. Incredibly slow, barely standard compliant (actually, there are a bunch of annoying bugs). And if you want calendar integration (arguably the best feature in Exchange) there is no good way around MAPI (at least not with Exchange 2007).

And that’s where open source email clients really fail. The only one with even attempted MAPI integration is Evolution. And that is one of the weakest parts of Evolution. Extremely unstable, slow, and so frequently flat out broken that I cannot really suggest using it for day to day work. Emails disappear, or their envelope is there and no content, parts of the headers are missing. The calendar is completely hit or miss: the latest version seems to get my single-instances meetings correct if they come from another user, get the time zone wrong if I enter them myself on the Blackberry or via OWA, and seems to completely miss out recurring meetings that were NOT entered by me. Not useful if I need to be able to rely on my calendar being correct (which is, after all, the point of a calendar).

So… what I do today is offlineimap to get emails from Exchange (or any other IMAP server) into a set of local MailDirs (this hides the latency of the IMAP implementation – especially important with Exchange), then Evolution to read that email locally and OWA for calendar.

Really, not a good solution at all. We need a decent MAPI client. The libraries are all there, the communication with the Exchange server is relatively easy to set up. What’s missing is an acceptable front end that can deal with the typical mess of email that people get (Thunderbird seems to be a good start and appears reasonably active and well maintained), that can do calendaring (again, Thunderbird with Lightening could do the job) and that has a reasonable UI, good keyboard shortcuts for the power users and most importantly is fast. So I guess we need MAPI integration into Thunderbird. Any takers?

This is a topic that I have been tossing around for a while. The fact that I gave a presentation about this at this week’s SambaXP conference in Göttingen has forced me to put some structure around my thoughts.

Single sign on is a commonly listed goal in the IT industry. You authenticate once and then have access to all kinds of applications or services via the net. This sounds good, but of course it comes at a price. The damage that can be done if your credentials are compromised increase dramatically if they can be used in many places.

A simple example. Let’s say you use your login credentials to also be able to access your email account. That’s wonderful – one less username and password to remember. But unfortunately a number of email clients (or email retrieval apps like getmail or offlineimap) have no convenient way to securely store your credentials – or are easily fooled into handing these credentials to a proxy server. Which suddenly exposes your “general pass key” to your account to an attacker.

Of course you can use SSL encryption on your email protocol (e.g., imaps) to make man in the middle attacks harder – but that only works if you have signed certificates and a correctly built and trusted CA in place with the CA certificates installed on all clients. Which gets a little harder with the proliferation of mobile clients. For example, when using your favorite email client on your Android phone (or Nokia N71, or…), who hasn’t clicked ‘accept’ when asked to verify the authenticity of an SSL certificate provided for the server that wasn’t signed with a key that’s installed in the client’s CA keychain. You may have looked at the certificate to make sure that it looked sane – but did you verify it?

That’s a huge risk when allowing authentication with typical username / password based single sign on credentials on internet-facing servers. Yet that’s a very common practice.

So what can you do? One school of thought is “just don’t do it”. Have separate credentials for all services, force distribution of CA certificates, enforce SSL or SSH as a minimum requirement to connect to any services and basically device and conquer the risk. But frankly, that makes mobile clients far less attractive, interaction with network facing applications and services more difficult and generally reduces productivity. And in the end, if users are forced to use too many distinct username / password combinations they tend to use simple algorithmic passwords (or simply write them down somewhere in clear text).

Another alternative is to look at ticket based systems like kerberos and the way they handle credentials. Kerberos (and it’s implementation in Microsoft’s Active Directory) encrypt all traffic and use a rather smart system to prevent man in the middle attacks. Assuming strong passwords (to prevent the well documented offline password guessing attacks) you can create a decent sign on system that can be used with mobile clients – assuming the client software stack includes the necessary code to authenticate against A/D – which unfortunately is not universally the case today.

Ideally for each account with a service provider you as the user would be able to pick how to authenticate – using a ticket authority of your choice – potentially with different identities between different services, potentially with the same. This way you could control which services share the same ID, ensure that all authentication is secure and at the same time make it easier to manage these identities securely in a mobile device.

Samba is a widely used open source implementation of the necessary pieces of A/D. It allows clients to authenticate against Active Directory servers or other kerberos based authentication servers and then only distribute tickets (that don’t include the actual credentials) to authenticate against services. One downside of using Samba’s model of implementing the different components of MSRPC as monolithic applications instead of APIs is that it makes it harder to use in this context (providing authentication services in the mobile world); also, Samba has gaps in its support for the full breadth of A/D (missing some of the SSPI providers).

Still, it’s a good start. An alternative could be Likewise Open as it matures. Kirshna and his team certainly are focused on a complete implementation of the necessary APIs – but the mobile client isn’t their focus area, either.

The discussion after my presentation showed some interest in the community to tackle the problem, but of course this will require the service providers and the device vendors to cooperate as well. But first more people need to understand the underlying problem that needs to be solved…

15 years ago today Linus released version 1.0 of Linux. It had been about two and a half years in the making: version 0.01 was released in August ’91 – I didn’t get started until a couple months later in Decmber ’91 with version 0.11 (0.10 which I tried the month before did’t like my 386sx/16). It’s so funny to see my old uni-wuerzburg.de address in that announcement… that’s been like three lifetimes ago.

The road to version 1.0 was longer than we thought… 0.99 went all the way to ‘patch level 15′ before Linus finally felt things were ready to be called 1.0. The version naming methodology back then was a little… archaic – things like Linux 0.99pl14.r. It’s much easier these days – Linux just released 2.6.29-rc8…

Too bad he missed out on releasing 2.6.29 on the anniversary. But then, dates like this are random and uninportant.

Excellent post by my friend Larry Augustin on the need to stop using the patent encumbered FAT filesystem.

I usually don’t just post links to other posts here, but I think this one is important. We do indeed act as if FAT wasn’t patent encumbered. And this touches both of my biggest hobbies – Linux and photography.

We as an industry (and collectively, as consumers) need to stop relying on this rather mediocre filesystem. There are much better choices available. Larry suggests ext2. That’s under GPL – maybe that’s an issue for some people. But there are really good BSD licensed filesystems out there. Without having done much research on the issue, maybe UFS would be a better choice? A lot of common operating systems support their own version of it already – so this might be a great starting point.

The industry would need to get together and standardize one standard version of it, but since everyone would have to give up something of their proprietary flavor, maybe that wouldn’t be so hard (and again, it would certainly be easier than dealing with a GPL licensed filesystem).

Update: Arjan pointed out that I was too quick in discounting Larry’s suggestion. There are in fact a number of BSD/MIT license style implementations of ext2 – there’s even a version for Windows and the Mac. So maybe ext2 is the best starting point after all… doing some searching around it seems that few of these projects are actively maintained, but that should be something the open source community could easily tackle…

Linux Kongress is the oldest Linux event. How do I know? Well, in 1994, Linux Kongress in Heidelberg was the first ever conference on Linux. It was a really cool event that brought most of the key Linux developers of the time together – many of us met there in person for the first time! And since then, every year there has been a Linux Kongress (okay, that’s mildly cheating, last year’s event was only held “in spirit” as part of linux.conf.eu when the Kernel Summit came to Cambridge and the Linux Kongress organizers didn’t want to try to create an event competing wiht that).

So yesterday I had the honor to be the closing presenter at the 2008 edition of Linux Kongress, after having to miss attending a few of the last years. It was great to see so many familiar faces and my only regret was that based on some personal travel that I did earlier in the week I had to miss the first day of the event. Still, I had a great time and enjoyed the opportunity to talk about “Mobile Linux” and what I think it will take for the community to create a really compelling OS for the mobile internet user. I tried to explain where Linux falls short at present and what we are doing with the Moblin community to create the technologies to help to close that gap. Of course I took a chance to show off the amazing five second boot of an EeePC. But see for yourself. The talk should be up in the archive of Linux Pro Magazin’s Online Conference Streaming, soon.

I’m a huge fan of the open source development methodology. But over the years I also had to realize that just because something is open source that doesn’t mean that automatically the right things happen. What is required is also good governance of the project, a strong community and the right leadership.

Many projects today have broad support from corporate contributors. The latest data from Greg KH shows that around 80% of the people contributing to the Linux kernel work for corporations and not just “on their own”. Some projects are more or less identified with corporations (MySQL or Clutter, for example). Others are mostly driven by key developers who are very independent – regardless where they work (Perl comes to mind).

So corporate involvement doesn’t seem to be an indicator for the success of open source. But the quality of the code that is being developed certainly seems to play a major role – if you look at the examples that I have given, that’s the common denominator. Good software, available in open source, appears to be one key aspect that is needed to create this virtuous cycle. The fact that the code is well written indicates that there is strong and smart leadership. That attracts more developers who want to participate. Now add good governance, i.e., you don’t turn away contributors (like apparently some projects are doing), you invite them to join you. You invite them to take major roles, to influence the direction. Yes, this will create disagreement and friction and maybe the project will move in directions that you didn’t initially have in mind. But it will also create a project that is vibrant and healthy and progressing at a fast pace.

Many times the projects will be able to deal with new ideas and different directions internally (there are tons of examples for that, basically every single one of the large open source projects with diverse contributors has gone through a number of revolutions driven by a new influx of developers – Gnome, Apache, Perl, even the Linux kernel). Sometimes this causes a fork (and often the original project atrophies – XFree86 and X.Org is an example here that I am very familiar with), sometimes after a time of forked co-existence the two project merge again (gcc and egcs). And sometimes this causes two healthy projects that are competing with each other and develop independently (or even three; look at FreeBSD, NetBSD, OpenBSD).

Influx of new ideas is good. The ability to absorb new ideas, to embrace people who want to change your project and who may disagree with you on the quality or direction of the existing code base is an important part of what gives open source software the opportunity to be better.

I’m thrilled to see commercial companies contribute to open source. That’s the life blood of many large projects. And I am thrilled if a company has the guts to realize that a project that is out there needs a major influx of new ideas and is willing to go out and contribute. Even if that sometimes ruffles some feathers. I have encouraged Likewise for a while to go out and make Samba better. So I was very happy to see that Krishna today announced Likewise Open Fall 08. It’s a project that complements and partly replaces Samba. That makes it easier for Linux (and other Unix-like OSs like Solaris or OS X) to be an equal player in a Windows (and Active Directory) environment. This is extremely well written code that fulfills a real need. Let’s hope the Samba community embraces it and uses it to make Samba an even stronger project. When I spoke last year at Samba XP that was what I tried to encourage them to do. And most of the developers (with a few notable exceptions) seemed to like the idea. I can’t wait for next years conference to see what happens.

I’m at GUADEC in Istanbul. I just listened to Leisa Reichelt talk about User Experience Design. Very nice presentation about something way too few open source developers really and truly focus on.

User experience design is very different from focusing on usability (and even that often isn’t done enough). User experience includes how people feel about using a product. So suddenly being cool can be a plus. The usability of the iPhone virtual keyboard is rather bad if you have larger hands. But the iPhone is considered way cool, so in the end the user experience might be good. On the other hand you can design a very usable piece of software, but if its underlying design is flawed and doesn’t reflect what the user really wants or needs to do, then the best user interface in the world is not going to fix it.

The open source community certainly is a good example for how things can go wrong with user experience. I brought a Linux laptop to the conference and after having used a Mac as my main system for so long it is really eye opening to me how many things just don’t work smoothly on a Linux system – and how the experience for me as the user is quite frustrating. From connecting to a wireless network to inserting an SD card from my camera – almost every application as a different design for its user interface and often they appear to be in the way of what I actually want to do.

Part of this is caused by the way open source software is written – a lose collection of mostly volunteers interested in creating good software. Upfront design is often considered in the way of creativity. But it also is an indication that this is a very hard problem. It takes a lot of research and hard work to truly understand what a user really wants. Let’s hope that some of the people in the audience are considering this as something worth focusing on.

I think I’ll call it “Jon Williams’ Law of Open Source”. But maybe that wouldn’t be fair as others have pointed this out before (including R0ml)…

Basically the point Jon was making at the end of his keynote was that the only way customers will continue to pay for open source software if the open source project that they are paying for doesn’t mature. Or in other words, if your company has an open source based business model, keep breaking the software and people will keep paying you.

Depressing