Community Matters

As a commenter on my previous post on this topic has pointed out, the same problem (the system stalls when due to Ext3′s bad design an fsync causes a complete sync of the filesystem, which on MLC-based SSDs causes long write stalls which the fsync blocks for) happens with Evolution as well. And of course running both Evolution and Firefox (a very typical scenario) it gets even worse.

I’ve worked with Arjan on improving LatencyTOP some more in order to track down these stalls. LatencyTOP now has a special mode (together with a matching kernel patch) that allows you to monitor all fsyncs with the file name that they are waiting for. Extremely useful in this context. What becomes obvious is that while we can’t fix the underlying problem with the bad combination of fsync-sync-MLC-stall without switching to a different filesystem, we can solve some of the other performance problems that are caused by the i/o system (and especially the filesystem) getting overloaded.

Basically the situation on the EeePC exposes another flaw of the combination of Ext3 and the new i/o scheduler. The way the priority calculations are run, the kernel journal daemon is getting starved, especially if writes stall. Thankfully there’s a way to improve the situation a little:
for i in `/sbin/pidof kjournald`; do
  /usr/bin/ionice -c1 -p $i;
done
This forces all instances of kjournald to run in the real time scheduling class. Which gives them more access to the very limited i/o bandwidth to the flash. Ideally put this somewhere where it is executed at boot time – something like /etc/rc.d/rc.local.

This helps in normal use – the system doesn’t stall as often, for example when just browsing the web. It doesn’t help with the massive stalls that are caused for example when you quit Evolution and it fsyncs all of the local copies of IMAP mailboxes.

We’re still looking into ways to improve that behavior (hoping for BTRFS to get into 2.6.29 is one of those strategies…).