Community Matters

…and other systems with cheap and slow SSDs like the Dell Inspiron Mini 9 or Acer Aspire One.

Important note: this is not written for ultrafast SSDs like the Intel SSDs found in high end laptops and servers; it’s targeting the type of SSDs usually found in netbooks. And of course it also doesn’t apply to hard disk based netbooks.

The problem? These very cheap SSDs tend to have terrible write performance. And filesystem designers apparently never considered the impact of devices that have fabulous access times, decent read speed but that come to a screeching halt when sending them too many writes at a time.

Most Linux distributions by default install ext3. Whether or not a journaling filesystem is a good idea for an SSD appears to be still up to debate; but unless the SSD does decent wear-leveling I would stay away from simple journals like what ext3 provides which keep overwriting the same part of the ‘disk’ over and over again.

Also, by default Linux tracks modifications to the access time of every inode. So every read from disk also creates a write updating the atime in the inode. To make things work better on a cheap SSD it’s best to turn this off using the mount option noatime.

The third problem you keep running into is the frequent fsync issue that I’ve described earlier. Here your best bet is a very current Linux distribution that has the latest versions of some of the applications that use fsync more heavily. This is certainly an area where Moblin is trying to be careful.

The fourth problem is the apparently common assumption that a Linux system has to have a swap partition. As it turns out, swap partitions are actually rather counter productive in a system as discussed here. As your system gets more heavily loaded (more things are running and competing for the limited write bandwidth to the SSD) you are now adding the Linux kernel as another creator of write accesses to the SSD – as it is trying to swap out pages to free memory. It’s much better to rely on Linux paging algorithm to keep fewer read only pages in memory – these are read back from “disk” when they are needed again, but as they never get modified they don’t need to be written out to the SSD. Yes, this means that you may run out of memory sooner – but if you do, that may be a sign that you are trying to do things that the little netbook wasn’t designed for (don’t run VirtualBox on it – be careful how many office and email applications you have open at the same time – and yes, limit the number of tabs in Firefox and restart it every once in a while.. Firefox is a real memory hog).

So what do I recommend? Long term it’s BTRFS. I am using this for some tests on one of my EeePCs and the difference is stunning. But BTRFS is far from being stable enough for real life and isn’t (and won’t be for some time) used in any Linux distributions.

In the meantime I’d go with ext2 with noatime and no swap partition. If your system (like most EeePCs) has two separate SSDs with different write speed, make sure that directories that are frequently written to (or fsync’ed) are located on that drive. I have explained how to do this in my posting on solving the Firefox and Evolution fsync issue.

Oh, I’ve played with JFS, XFS and yes, even vfat as possible alternatives. Neither of them provides any noticeable advantage over ext2 – and since they are less frequently used I am concerned about potentially running into bugs and problems that haven’t been discovered by other users, yet.

As of a few days ago, a new brand of third party SSDs is available for the EeePC. The RunCore SSDs are now available at MyDigitalDiscount. I’ve already ordered two and can’t wait for them to arrive. I’ll post benchmarks, soon.

It seems a combination of several of the earlier ideas seems to do the trick. If you came to this post directly, here are the earlier posts for some context::

• EeePC Hangs with Firefox
• More on the EeePC hangs

What seems to work fairly well for me is to run the ionice script at boot time to speed up the overall behaviour of Ext3 (especially on the slow MLC drive of the EeePC – you can find that in the second article mentioned above), but to also move the .evolution and .mozilla directories from your home directory (and any other directory that regularly receives fsync calls – use LatencyTOP to find out which ones those are) to a separate partition on the SLC drive and to format that partition with Ext2 (and not Ext3).

Now this means that this workaround is going to be rather hard for people running the default Xandros installation on the EeePC – that one uses all of the fast SLC drive (/dev/sda) for the unionfs that allows it to easily restore its original settings. Since I don’t run Xandros I can’t really help with how you could modify it to make this work.

If you have installed a different version of Linux on your flash based EeePC, then this should be fairly doable. Most likely /dev/sda contains your /boot (if separate) and your / filesystems. For data integrity reasons you may not want to switch either of them from Ext3 to Ext2 (as you lose the journal doing that). So instead I suggest to shrink the / filesystem and create a new partition in the remaining space, to format this with Ext2, mount it under something like /disposabledata (remember, if your system crashes, there’s a higher risk that you lose the content of this filesystem) and then move ~/.mozilla and ~/.evolution to that filesystem. The risk you are taking isn’t really that big, if you think about it. You may lose your browsing history. Or you may lose local copies of email that are stored on the server, anyway. What would be annoying to lose are the stored passwords and the overall settings of both firefox and evolution – but a little cron job that regularly copies that data back into different directories in your /home directory should do the trick.

Details will be different depending on your scenario, but here is the rough outline of what I did and which seems to be working well for me so far:

• shrink the / partition on /dev/sda. There are many tutorials how to do that, I used this one to guide me through the process. How big that partition needs to be depends on how big your ~/.mozilla and ~/.evolution folders are. Some of the popular distributions by default don’t use partitions but LVM volumes to hold your filesystems. Again, I don’t use that, so I can’t provide you with detailed instructions how to shrink the filesystem in there and create a separate one as Ext2 filesystem – but it can’t be that hard to figure out.
• create a new partition in the now available space on /dev/sda and format it with Ext2 (usually that’s done with mkfs.ext2)
• modify /etc/fstab to mount this to an appropriate spot; I like to make it clear from the name of the mount point that this is not a journalled filesystem, so the line in my /etc/fstab looks something like this:
  /dev/sda3 /disposabledata ext2 defaults,noatime 0 0
  Obviously you need to use the right partition that you create earlier – it was /dev/sda3 in my case.
• still as superuser, create a directory for your user on that filesystem and chown it to that user:
  mkdir /disposabledata/user; chown user /disposabledata/user
• as your regular user quit firefox and evolution and then move their respective directories to that directory:
  mv ~/.evolution ~/.mozilla /disposabledata/user
• create links from your home directory to these new locations:
  ln -s /disposabledata/user/.evolution ~
  ln -s /disposabledata/user/.mozilla ~
• finally, create a little cron script that copies these directories back to your home directory at a regular interval – ideally keeping one generation of backups so that regardless when your system crashes, you always should have one intact copy of this data. Here’s what I do:
  #!/bin/bash
SRCDIR=/disposabledata/user
TARGETDIR=/home/user/backup
SUBDIRS=".mozilla .evolution"
# sanity checks
if [ -e $TARGETDIR -a ! -d $TARGETDIR ] ; then
   echo $TARGETDIR is not a directory
   exit 1
fi
[ -d $TARGETDIR ] || mkdir $TARGETDIR
for i in $SUBDIRS; do
   [ -e $TARGETDIR/$i.bak ] && rm -rf $TARGETDIR/$i.bak
   [ -e $TARGETDIR/$i ] && mv $TARGETDIR/$i $TARGETDIR/$i.bak
   cp -r $SRCDIR/$i $TARGETDIR
done

And in the unfortunate event that this happens and your /disposabledata filesystem gets corrupted, simply recreate that filesystem and copy the last version of the backup folders back to that spot. You may have lost some of the browsing history and potentially some recent changes that you made, but most of your settings and data should still be around.

But most importantly, the system will feel much more usable with significantly fewer stalls and hangs. It certainly does for me.

As a commenter on my previous post on this topic has pointed out, the same problem (the system stalls when due to Ext3′s bad design an fsync causes a complete sync of the filesystem, which on MLC-based SSDs causes long write stalls which the fsync blocks for) happens with Evolution as well. And of course running both Evolution and Firefox (a very typical scenario) it gets even worse.

I’ve worked with Arjan on improving LatencyTOP some more in order to track down these stalls. LatencyTOP now has a special mode (together with a matching kernel patch) that allows you to monitor all fsyncs with the file name that they are waiting for. Extremely useful in this context. What becomes obvious is that while we can’t fix the underlying problem with the bad combination of fsync-sync-MLC-stall without switching to a different filesystem, we can solve some of the other performance problems that are caused by the i/o system (and especially the filesystem) getting overloaded.

Basically the situation on the EeePC exposes another flaw of the combination of Ext3 and the new i/o scheduler. The way the priority calculations are run, the kernel journal daemon is getting starved, especially if writes stall. Thankfully there’s a way to improve the situation a little:
for i in `/sbin/pidof kjournald`; do
  /usr/bin/ionice -c1 -p $i;
done
This forces all instances of kjournald to run in the real time scheduling class. Which gives them more access to the very limited i/o bandwidth to the flash. Ideally put this somewhere where it is executed at boot time – something like /etc/rc.d/rc.local.

This helps in normal use – the system doesn’t stall as often, for example when just browsing the web. It doesn’t help with the massive stalls that are caused for example when you quit Evolution and it fsyncs all of the local copies of IMAP mailboxes.

We’re still looking into ways to improve that behavior (hoping for BTRFS to get into 2.6.29 is one of those strategies…).

I’ve done some more work on figuring out why my EeePC literally freezes for a second and more at a time when I work with Firefox. I used LatencyTOP to track this down and think that I have put together the pieces of this puzzle.

The reason this hits me especially hard is a combination of several independent issues:

1. Firefox 3 uses sqlite which aggressively uses fsync to make sure that the on disk database of websites visited stays in sync. The corresponding bug is marked as resolved, fixed but I would consider that a serious overstatement.
2. Ext 2 and 3 have a bug where an fsync on a file actually forces a sync on the disk. This, too, has been discussed before.
3. Many MLC SSDs have a problem with many small writes at once (this was discussed in extensive detail in this great AnandTech article).
4. Finally, as I mentioned before, there appears to be something wrong with the MLC drive in my EeePC 1000 (I’m still waiting for the 32GB SLC drive (and, just to do comparisons, a 64GB MLC drive) from MyDigitalSSD). And since I have my complete Fedora installation on the secondary drive, all writes to temp files and log files add to the write delays.

It gets so bad that LatencyTOP show the system stalled for 1500ms just waiting for one Firefox fsync to complete.

So what can you do to fix this if you run into the same issue (which seriously hampers the usability of an affected system)? Well, you could put your /home directory on the SLC (/dev/sda) in your EeePC. But that’s not really a good solution as most of us want to use the large drive in order to have space for all of our files.

You could switch to a different filesystem. But Fedora (and many other current distributions) are very much built around the assumption that you are using Ext 3. And it seems that ReiserFS isn’t any better in that respect. The filesystem that will fix the problem, BTRFS is still not quite ready for real life deployment.

Finally, you could put a small partition on your SLC drive, and move your .sqlite files (in the .mozilla/firefox/randomlettersprofile directory) onto that drive. I tried that and it didn’t help, either.

So I’m still searching for a solution.

Update: I think I have at least a partial solution.

Several people have asked me in the last few days about a comparison of the EeePC 900 and the EeePC 901. And the more I try to find an easy response, the more I realize just how complicated this has gotten.

Let’s start from the outside. The 900 and 901 models are very similar, they have the same screen, they both share the same keyboard (so they both have the keyboard issue that made me decide to switch from my EeePC 901 to an EeePC 1000 a few weeks ago), but there are a few differences. The 901 has the “new” body which connects the screen with metal hinges to the case, the 900 uses plastic hinges and is overall more similar to the 701. The 901 case feels a little sturdier and well built, but that may be a perception issue; it is 10% heavier than the 900. The two models are NOT exactly the same size, so be careful when looking at sleeves, etc.

Now let’s look at the internals. Admittedly there are way too many versions of the EeePC for anyone to keep track of, so I’ll try to get some structure in this. The main difference is of course the CPU – the EeePC 900 is based on an Intel Celeron M353 processor, the EeePC 901 is based on an Intel Atom N270 processor. The latter provides better performance and lower power consumption. On top of that Asus is shipping a larger battery with the 901, the net total of all this is about 2x the battery life compared to the 900.

Also, the 901 is the only one (in the 9xx series) that supports 802.11/n wireless, the other ones only support 802.11g.

There are a number of different storage options available. They seem to keep changing, but last time I checked you could get the both the 900 and 901 with 12GB SSD (that’s two separate devices, a 4GB system drive and an 8GB user data drive) and Windows or at the same price with 20GB SSD (4+16) and Linux. Lately the 900 also comes as a 16GB SSD model with just one built in SSD – this one with different prices for Windows and Linux versions. Be very careful with this model (often called EeePC 900 16G). I have seen a number of reports that the SSD in this model shows the same serious write performance degradation that I have observed on the 32GB secondary SSD in my EeePC 1000; I’m still trying to track down details, but if you have this as your system drive (where temp files and log files go) you may be in for a very sluggish user experience.

All of them come with 1GB of RAM, but that’s trivial to upgrade to 2GB (compare that to the Acer Aspire One where it is the ridiculously difficult to upgrade the memory). I highly recommend that upgrade as it really helps day-to-day performance when using these little computers.

Oh yes and then there’s the 900A (which brings the Atom processor to the 900 series body, but seems to make some other compromises, for example a lower resolution web cam) and the 904HA and 904HD (these ones have a 160GB HDD). I never owned either of them, so I can’t say too much about them.

I mentioned before in my article comparing the EeePC 901 with the 1000 that the keyboards of some Netbooks leave room for improvement. Today at IDF in Taipei I spent some time looking at the current crop of Netbooks and I realized that this is an endemic issue. Yes, there isn’t much space and therefore vendors have to make some compromises, but where these compromises impact day to day usability of the resulting computers, I think people are making a mistake.

The Asus EeePC 901 / 901 have the two-thirds sized ‘;‘ and ‘'‘ keys which make you hit ‘enter‘ when you want to hit ‘'‘. From what I could tell in the IDF showcase, the Netbooks from Founders, Lenovo, Toshiba and Dell share this unfortunate design. The only Netbooks that I found with regular keyboards were the Acer Aspire One and the EeePC 1000 / 1000H.

Even worse, the Netbooks from Dell, Sylvania and Lenovo had keys moved around to make their keyboard fit into the available space. Typing fast and blind on these systems will require a lot of getting used to – and jumping between these and standard keyboards is likely to be impossible. On several of these keyboard I had to stare long and hard to find keys like ‘~‘ or ‘\‘ on either side of the space bar (Sylvania) or ‘'‘ between the Windows Menu key and cursor left key (Dell) or the ‘]‘ not next to but below the ‘[‘, actually between the ‘'‘ key and enter.

I question the wisdom of these layouts…

The touchpads of the newer EeePCs (certainly the 900/900A/901/1000/1000H) all support some form of multitouch – not all the gestures we’ve seen on other pads, but the most important ones like scroll. The maker of the touchpad (Elantech) has posted newer Windows drivers for the touchpad a couple of months ago which according to several reports dramatically improves the smoothness of the touchpad’s response and add many more gestures (like pinch, browser navigation, etc.), but under Linux the support still isn’t really ideal (and no drivers from Elantech appear to be available). Especially the two-finger scroll that I am so used to from my Mac often doesn’t work. It scrolls a little, but then suddenly thinks that you are actually dragging one finger and switches to select mode – very frustrating.

Here’s an interesting little fact that I just ran across. It seems that the two flash drives (or SSDs) in my EeePC 1000 don’t have the same performance characteristics. I was trying to track down some surprising performance (and boot time) discrepancies on my EeePC 1000 and ran bonnie++ on different partitions in my system. Here are the results:

The smaller first sdd (/dev/sda under Linux) gets around 12MB/s write performance, whereas the larger second sdd (/dev/sdb) gets only around 3.5-4MB/s write performance. The read performance between both devices seems to be very similar.

Certainly something to be aware of.

More interestingly, while looking for further information on this issue I ran across the MyDigitalSSD’s 32GB SLC SSD for the EeePC which promises to significantly speed up both read and write performance on the second SSD. Using these benchmarks (unfortunately done under Windows, so not really comparable with the Linux results posted here) this SSD is 50% faster for reads and twice as fast for writes compared to the faster results that I got for the first SSD in my EeePC. Which should give me 6-8x the write performance for my larger SSD.

I’ll order one and report about the results.

Linux Kongress is the oldest Linux event. How do I know? Well, in 1994, Linux Kongress in Heidelberg was the first ever conference on Linux. It was a really cool event that brought most of the key Linux developers of the time together – many of us met there in person for the first time! And since then, every year there has been a Linux Kongress (okay, that’s mildly cheating, last year’s event was only held “in spirit” as part of linux.conf.eu when the Kernel Summit came to Cambridge and the Linux Kongress organizers didn’t want to try to create an event competing wiht that).

So yesterday I had the honor to be the closing presenter at the 2008 edition of Linux Kongress, after having to miss attending a few of the last years. It was great to see so many familiar faces and my only regret was that based on some personal travel that I did earlier in the week I had to miss the first day of the event. Still, I had a great time and enjoyed the opportunity to talk about “Mobile Linux” and what I think it will take for the community to create a really compelling OS for the mobile internet user. I tried to explain where Linux falls short at present and what we are doing with the Moblin community to create the technologies to help to close that gap. Of course I took a chance to show off the amazing five second boot of an EeePC. But see for yourself. The talk should be up in the archive of Linux Pro Magazin’s Online Conference Streaming, soon.