Benchmarks using a SSD to speed up ext4 with external journal and enhanceio

I’ve just made some tests to see how EnhanceIO performs and to see how data=journal and data=ordered perform when using a SSD as external journal device..

Tests have been made on the following hardware

  • CPU: Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz
  • SSD: SAMSUNG MZ7PD120HAFV-000DA (120G)
  • MD0: Software Raid 1 over a partition (sdb3 and sdc3) on the following two HDDs.
  • HDD1: WDC WD1003FBYZ-010FB0
  • HDD2: Seagate Constellation ES.3
  • RAM: 16 GB
  • OS: Linux nyota 3.16.0-4-amd64 #1 SMP Debian 3.16.39-1+deb8u1 (2017-02-22) x86_64 GNU/Linux, Debian Jessie

Test #1

Mount options: noatime,user_xattr,errors=remount-ro,commit=300,journal_checksum,journal_async_commit,data=ordered, Journal Device Size 1GB

Enhance IO: Policy lru, Mode read-only, Size 8GB

writes rewrites random reads mixed workload random writes pwrites fwrites
min
throughput
per thread
42980.63 kB/sec 34438.25 kB/sec 108361.57 kB/sec 75899.74 kB/sec 31737.90 kB/sec 42651.92 kB/sec 59742.36 kB/sec
max
throughput
per thread
43066.30 kB/sec 66467.18 kB/sec 109245.78 kB/sec 235433.14 kB/sec 67148.04 kB/sec 42739.32 kB/sec 59996.75 kB/sec
avg
throughput
per thread
43023.47 kB/sec 50452.71 kB/sec 108803.68 kB/sec 155666.44 kB/sec 49442.97 kB/sec 42695.62 kB/sec 59869.55 kB/sec

Test #2

Mount options: noatime,user_xattr,errors=remount-ro,commit=300,journal_checksum,journal_async_commit,data=ordered, Journal Device Size 1GB

Enhance IO: Disabled

writes rewrites random reads mixed workload random writes pwrites fwrites
min
throughput
per thread
41,44 MB/s
(-0,52 MB/s)
33,17 MB/s
(-0,45 MB/s)
113,23 MB/s
(+7,41 MB/s)
63,61 MB/s
(-10,50 MB/s)
41,84 MB/s
(+10,84 MB/s)
41,25 MB/s
(-0,39 MB/s)
58,85 MB/s
(+0,51 MB/s)
max
throughput
per thread
41,48 MB/s
(-0,57 MB/s)
66,40 MB/s
(+1,49 MB/s)
118,59 MB/s
(+11,91 MB/s)
72,21 MB/s
(-157,69 MB/s)
55,35 MB/s
(-10,21 MB/s)
41,29 MB/s
(-0,43 MB/s)
59,60 MB/s
(+1,01 MB/s)
avg
throughput
per thread
41,46 MB/s
(-0,55 MB/s)
49,78 MB/s
(+0,51 MB/s)
115,91 MB/s
(+9,66 MB/s)
67,91 MB/s
(-84,10 MB/s)
48,60 MB/s
(+0,31 MB/s)
41,27 MB/s
(-0,41 MB/s)
59,23 MB/s
(+0,76 MB/s)

I divided the results with 1024 here to make the difference to the first test obvious. Differences of +- 1 MB/s can safely be ignored in my opinion. Also I couldn’t believe the -157,69 MB/s and ran the test another time with and without enhanceio. The results stayed same. So enhanceio seems to offer some improvement for specific workloads.

Test #3

Mount options: noatime,user_xattr,errors=remount-ro,commit=300,journal_checksum,journal_async_commit,data=journal, Journal Device Size 1GB

Enhance IO: Policy lru, Mode read-only, Size 8GB

writes rewrites random reads mixed workload random writes pwrites fwrites
min
throughput
per thread
41,75 MB/s
(-0,21 MB/s)
41,39 MB/s
(+7,76 MB/s)
3,78 GB/s
(+3,68 GB/s)
50,54 MB/s
(-23,58 MB/s)
36,12 MB/s
(+5,13 MB/s)
38,71 MB/s
(-2,93 MB/s)
56,63 MB/s
(-1,70 MB/s)
max
throughput
per thread
42,29 MB/s
(+0,23 MB/s)
41,67 MB/s
(-23,23 MB/s)
4,63 GB/s
(+4,53 GB/s)
6,82 GB/s
(+6,59 GB/s)
36,15 MB/s
(-29,42 MB/s)
39,38 MB/s
(-2,35 MB/s)
58,76 MB/s
(+0,17 MB/s)
avg
throughput
per thread
42,02 MB/s
(+0 MB/s)
41,53 MB/s
(-7,73 MB/s)
4,21 GB/s
(+4,10 GB/s)
3,43 GB/s
(+3,28 GB/s)
36,13 MB/s
(-12,14 MB/s)
39,05 MB/s
(-2,64 MB/s)
57,70 MB/s
(-0,76 MB/s)

I couldn’t believe these results, neither. Note that the bold values are in GB/s. So I issued a reboot and re-checked. After a Reboot I am presented with the same values. Trying with enhanceio disabled

Test #4

Mount options: noatime,user_xattr,errors=remount-ro,commit=300,journal_checksum,journal_async_commit,data=journal, Journal Device Size 1GB

Enhance IO: Disabled

writes rewrites random reads mixed workload random writes pwrites fwrites
min
throughput
per thread
41,84 MB/s
(-0,12 MB/s)
41,82 MB/s
(+8,19 MB/s)
3,45 GB/s
(+3,35 GB/s)
52,76 MB/s
(-21,36 MB/s)
36,46 MB/s
(+5,46 MB/s)
39,49 MB/s
(-2,15 MB/s)
54,88 MB/s
(-3,45 MB/s)
max
throughput
per thread
41,98 MB/s
(-0,06 MB/s)
42,05 MB/s
(-22,85 MB/s)
4,32 GB/s
(+4,22 GB/s)
6,83 GB/s
(+6,76 GB/s)
36,53 MB/s
(-29,04 MB/s)
40,10 MB/s
(-1,63 MB/s)
55,76 MB/s
(-2,82 MB/s)
avg
throughput
per thread
41,91 MB/s
(-0,09 MB/s)
41,93 MB/s
(-7,33 MB/s)
3,88 GB/s
(+3,78 GB/s)
3,44 GB/s
(+3,29 GB/s)
36,49 MB/s
(-11,78 MB/s)
39,80 MB/s
(-1,89 MB/s)
55,32 MB/s
(-3,13 MB/s)

So, while I am wondering about how to achieve that high numbers (mind the GB/s in the previous results) we can clearly see that there is an improvement when using data=journal combined with an external journal lying on a SSD over data=ordered. The difference between using EnhanceIO is not so clear. Let’s take another look at the information I gathered using EnhanceIO by taking a look at some min/max values of with and without:

min random reads max random reads min mixed workload max mixed workload
Without EnhanceIO #2 113,23 MB/s 118,59 MB/s 63,61 MB/s 72,21 MB/s
Without EnhanceIO #4 3,45 GB/s 4,32 GB/s 52,76 MB/s 6,83 GB/s
With EnhanceIO #1 105,82 MB/s 106,68 MB/s 74,12 MB/s 229,91 MB/s
With EnhanceIO #3 3,78 GB/s 4,63 GB/s 50,54 MB/s 6,82 GB/s

The difference between using EnhanceIO (if you compare #2 with #1 and #4 with #3) is not that obvious; I guess this highly depends on the work load and since my journal is on a SSD already, I would need a faster device for EnhanceIO to see a notable difference. From what I can tell now, it seems to help more to place the journal onto an SSD and use data=journal than using enhanceIO.

3 Comments

  • sr

    18. März 2017 at 14:07 Antworten

    Thanks!

    Useful to know that ext4 external journal on SSD can be as useful as other caching technologies.

    1) Any reason you chose EnhanceIO over other possibilities? (lvmcache, bcache, FlashCache, …)

    (Especially since https://wiki.archlinux.org/index.php/EnhanceIO seems to indicate that it is currently:
    – broken: https://github.com/stec-inc/EnhanceIO/issues/106
    – unmaintained: https://github.com/stec-inc/EnhanceIO/issues/114)

    2) Could you describe more the tests you performed? Is it a benchmark?

    • jean

      18. März 2017 at 22:22 Antworten

      Hi there,

      I haven’t used lvm/dm-cache here since that looked difficult to set up at a first glance (just set it up on two servers and seems to work fine, haven’t looked at the performance, yet). Bcache is actually my favorite, but requires you to setup a special data partition/block device. So that’s not an easy drop-in for an already working system. I won’t use FlashCache due to all its dependencies; compiling it is a nightmare. From all those techniques EnhanceIO (mind that I didn’t take a look at flashcache since my compiling tries and getting all the required dependencies was driving me crazy) is actually extremly easy to use because it does work transparent on an already existing filesystem without the need to touch that filesystem.

      But yeah, you’re right. EnhanceIO looks pretty much dead which is why I took a closer look at dm-cache lately. Though from my tests the most impressive results can be achieved by simply using an external journal.

      To answer your second question: Yeah, a few days later I noticed that I provided benchmarks without writing down how. Sorry for that. I used the same command as shown here: https://raid6.com.au/posts/fs_ext4_external_journal/. So… I used iozone. I think I’ve played with -r and -s.

      • sr

        19. März 2017 at 2:18

        lvmcache (LVM in general) is difficult because of the multiple commands, options, and layers.

        I tested it via blogs (*) and careful reading of man pages (this was my own, real unbackuped data), but I encountered some bugs, so I deinstalled it.

        A developper on irc has been really helpful, though, if I had known, I would have ask here for advices, before testing.
        This was with a random version of debian/jessie-backports (I prefer not to compile these softwares), this may explain the bugs.

        Maybe lvmcache is/will be better on debian/stretch.

        Like ext4 external journal, lvmcache/dm-cache is reversible and can be added (and removed) onto existing filesystem.
        I did not benchmark it, but I did not really „feel“ the additional speed (my setup may be special, I am on NFS)

        With external journal, my desktop feels more responsive, so thanks again for pointing this out 🙂

        iozone GB/s results looks suspicious: this looks more like RAM speeds rather than SSD speeds.
        But this is convincing for external journal vs. EnhanceIO (or other cache methods).

        (*) for example http://blog-vpodzime.rhcloud.com/?p=45

Post a Comment