November 09, 2016
Howto optimize Linux – SD card as rootfs
The write performance of a SD card is a little bit different then using a conventional spinning disk under Linux. The main problem is that a SD card can not write data over existing data. So whenever it want to write something it has to delete the content of that section first. To add to that initial problem, a SD card can only delete certain sizes of blocks, commonly between 2 Mb and 16 Mb depending on the SD card. So even if you have a small write cycle of let’s say a couple of Kb’s, the SD card actually has to read everything from it erasure block into memory, delete that section and then write back the altered section back onto the SD card.
Knowing this, we might want to do the following two things;
- Minimize any write cycles to the SD card
- Align the partition to the sice of the SD card his erasure block to prevent it sometimes have to delete/write two of those for most files.
Now let’s start with point 2 first as we might want to do that even before we start installing our linux system. We first need to investigate what the erasure clock size is of our SD card. The Linaro people wrote a small to to investigate that; “FlashBench“.
Download the tool, compile and run.
$ sudo ./flashbench -a /dev/mmcblk0 –blocksize=1024
align 134217728 pre 735µs on 1.08ms post 780µs diff 324µs
align 67108864 pre 736µs on 1.05ms post 763µs diff 300µs
align 33554432 pre 722µs on 1.04ms post 763µs diff 294µs
align 16777216 pre 727µs on 1.05ms post 772µs diff 302µs
align 8388608 pre 724µs on 1.04ms post 768µs diff 299µs
align 4194304 pre 741µs on 1.08ms post 788µs diff 317µs
align 2097152 pre 745µs on 950µs post 811µs diff 171µs
align 1048576 pre 745µs on 945µs post 807µs diff 169µs
align 524288 pre 743µs on 936µs post 799µs diff 165µs
align 262144 pre 746µs on 948µs post 809µs diff 171µs
align 131072 pre 737µs on 935µs post 804µs diff 165µs
align 65536 pre 735µs on 925µs post 796µs diff 159µs
align 32768 pre 735µs on 925µs post 800µs diff 157µs
align 16384 pre 745µs on 911µs post 781µs diff 148µs
align 8192 pre 785µs on 808µs post 725µs diff 53.3µs
align 4096 pre 784µs on 788µs post 779µs diff 5.85µs
align 2048 pre 787µs on 793µs post 789µs diff 4.65µs
As you can see above there is big jump in “diff” time (the last column) from 4191304 (4mb) and 2097152 (2mb). It almost split by half. Based in the example readme in flashbench, this indicates that there was no performance overhead reading two blocks over the 2mb boundary, but there was for 4mb boundary. The guess is then that the erasure block is 4mb large on this sd-card.
Similarly, both 8 KB and 4 KB boundaries are special. The logical explanation for this is that the card has 8 KB pages, but can use multi-plane accesses to read two 8 KB pages simultaneously. So to conclude on above;
- We want to align our partition table to the size of a 4mb erasure block.
- We want to format it with a blocksize of 4kb and the “stride” value should be 2. This will cause ext4 to think that units of 2 blocks (8k) can and should be treated as one.
- Ext4 should have the stripe-size set to 512. This value was calculated by taking 4M (guessed erasure block size) dividing by 8K (size of a stride, 2 times block size (4K)). This will (hopefully) cause Ext4 to try to align writes so that while erasure blocks are written continuously and make it avoid sub-block updates.
Simple, isn’t it? So if we want to start our partition aligned with a 4 mb erasure block size. Fdisk uses blocks of 512 bytes, so that means that we want to start at 4*1024^2/512 = 8192. If you want more partitions you can make use of multiple of that number, but keep in mind we start at 0 NOT 1. So if you want to have a FAT partition of around 16mb, we start at 8192 and we end that partition at (4*8192)-1 = 32767. Then the next partition will start at the exact start of a new erasure block again at 32768.
So now we want to format our partition with EXT4 and the above mentioned parameters. Let’s also disable the journalling to increase the performance using point 1 (minimise write cycles).
mkfs.ext4 -O ^has_journal -E stride=2,stripe-width=512 -b 4096 -L SDCardLinux /dev/mmcblk0p1
That would finalise point 2, aligning the partition(s) to the size of the erasure block and optimizing the EXT4 partitioning to make full use of it.
So what can we do to minimize the amount of write cycles while running our linux distribution? Let’s first make sure that linux is no longer updating the files with their access and opening times. (when you open a file that date/time is written to the file). We can prevent that by adding the “noatime” parameters to the “/etc/fstab”. Change it to something like this (as an example);
/dev/root / ext4 defaults,noatime 0 0
Secondly, we can chance the disk sheduler. The standard “CFQ” sheduler is optimized for spinning disk and tries to rearrange cycles such a way access times are shortened. In the case of a SD card access time are already multifold times faster then a conventional spinng disk, so let’s make sure al cycles are done immediately. We can change the sheduler in three different ways.
- Add linux boot parameter “elevator=noop” as kernel parameter
- By adding block/sda/queue/scheduler = noop to your /etc/sysfs.conf (requires the sysutils package)
- Compile the kernel using the “noop” sheduler as default. (This is the one I personally use now)
Now we can also move some commonly used linux folders to a tmpfs, especially the ones which we don’t care about to much, such as “tmp”. So let’s edit our “/etc/fstab” again and let add the following;
tmpfs /var/log tmpfs defaults 0 0
tmpfs /tmp tmpfs defaults 0 0
tmpfs /var/tmp tmpfs defaults 0 0
With above tweaks you should observe some speed optimization, but at least you minimize the wear of your SD card.