Boot times

In my last blog I wrote about optimizing the boot-times using runit and a special kernel. As of today I switched to systemd in Lunar Linux and I’m using connman for networking; Let’s take a look at the boot-times again…

Systemd already comes with some good time-tracking functionality:

wdp@localhost ~ $ systemd-analyze 
Startup finished in 4777ms (kernel) + 9265ms (userspace) = 14043ms

4,7 seconds for the kernel is quite much. My goal are 3 seconds for the kernel.

Kernel

In some distributions, and if you ever compiled the linux kernel yourself, you might have wondered about the setting CONFIG_PRINTK_TIME which is in: Kernel hacking —> [*] Show timing information on printks. This setting is especially useful, if you want to tune your boot-times because you can easily see where a lot of time is needed.

Let’s take a look at my kernel, by doing:

dmesg | less

You can see (if you would have my notebook) that we have some delays:

[    1.402408] yenta_cardbus 0000:0f:06.0: TI: mfunc 0x01aa1b22, devctl 0x64
[    1.704038] ata2: SATA link down (SStatus 0 SControl 300)

again:

[    1.773526] sd 0:0:0:0: [sda] Attached SCSI disk
[    1.896884] yenta_cardbus 0000:0f:06.0: ISA IRQ mask 0x0cf8, PCI irq 22

another one:

[    2.482557] drivers/rtc/hctosys.c: unable to open rtc device (rtc0)
[    3.209110] usb 7-2: new full speed USB device number 2 using uhci_hcd
[    3.312166] Synaptics Touchpad, model: 1, fw: 6.3, id: 0x12a0b1, caps: 0xa04713/0x204000/0x0

So basically you’re looking for long delays between the first line and the line:

[    3.565158] Freeing unused kernel memory: 552k freed

You can easily see, that the kernel is loading stuff which it wouldn’t need for just a simple boot. So let’s reduce that. The general rule is: Everything which is NOT needed to properly boot your system, tho needed later, should be a module. Of course, there are some exceptions. Imagine you’ve only got an usb keyboard and your system is broken / you’re booting without beeing able to load usb modules. Without another keyboard you might not be able to do something.

Let’s see what I can modularize (I’ll mention only those, which I didn’t have as module, and only the first half. going through the whole kernel config would let this post explode. As I already said: everything which is needed for booting Y everything which you need and can’t set modular Y, everything else you need M, everything you don’t need n)

  • Processor type and features —>
    • <M> /dev/cpu/microcode – microcode support
    • <M> /dev/cpu/*/msr – Model-specific register support
    • <M> /dev/cpu/*/cpuid – CPU information support
  • Power management and ACPI options —>
    • CPU Frequency scaling —>
      • <M> CPU frequency translation statistics
      • <M> ACPI Processor P-States driver
    • Memory power savings —>
      • <M> Intel chipset idle memory power saving driver
  • Bus options (PCI etc.) —>
    • <M> PCI Stub driver
    • <M> PCCard (PCMCIA/CardBus) support —> (yes, here’s the yenta stuff in, due to the M above, everything inside it will be M also)
  • Networking support —>
    • <M> RxRPC session sockets
    • <M> RF switch subsystem support —>
  • Device Drivers —>
    • <M> Connector – unified userspace <-> kernelspace linker —>
    • [*] Block devices —>
      • <M> Packet writing on CD/DVD media
      • <M> Virtio block driver (EXPERIMENTAL)
    • -*- Misc devices —>
      • {M} TI Flash Media interface support (EXPERIMENTAL)
    • [*] Network device support —>
      • {M} Generic Media Independent Interface device support
      • <M> Virtio network driver (EXPERIMENTAL)
      • Notice: I disabled all network cards I don’t have.

Again: this is just a small snippet; I did a bit more at the kernel.

Before:

Installation Info
  Installed Version: 2.6.39.3
  Last compile time: 47m30s
  Approx. Size:      39904KB
  Files Installed:   1391

wdp@localhost ~ $ du /boot/2.6.39.3-x86_64 -h
3.5M    /boot/2.6.39.3-x86_64

After:

Installation Info
  Installed Version: 2.6.39.3
  Last compile time: 26m34s
  Approx. Size:      31032KB
  Files Installed:   1218

root@localhost ~ # du -h /boot/2.6.39.3-x86_64 
2.9M    /boot/2.6.39.3-x86_64

Ignore the compile time; ccache was speeding the compile up a bit and the first compile was made while the notebook was in-use. However, you can see quite nicely that the kernel-size was reduced (by 0,6 MB). Let’s take a look at the boot-time:

wdp@localhost ~ $ systemd-analyze 
Startup finished in 2972ms (kernel) + 10028ms (userspace) = 13001ms

So I just improved the kernel-part by 1,8 seconds. We can try to reduce the boot-time of the kernel even more, by going more closely through the kernel config and de-select even more things which we wouldn’t need. You could also de-select various debug-features and printed informations (those are just slowing things down, and if you aren’t a kernel developer some (not all!) of those information aren’t useful for you anyway) – But such tuning is not the goal of my guide here. I already reached my goal of 3 seconds .)

Now it’s time to look at the other side of our boot-time.

systemd’s services

systemd-analyze has some very useful functionality:

root@localhost ~ # systemd-analyze blame
  3300ms hostname.service
  3222ms systemd-logind.service
  3195ms wpa_supplicant.service
  3187ms connman.service
  3179ms avahi-daemon.service
  1606ms systemd-vconsole-setup.service
   720ms udev-trigger.service
   681ms ConsoleKit.service
   400ms udev.service
   369ms dbus.service
   354ms var-lock.mount
   311ms media.mount
   303ms var-run.mount
   299ms systemd-user-sessions.service
   186ms systemd-sysctl.service
   159ms console-kit-log-system-start.service
   136ms irqbalance.service
   114ms systemd-remount-api-vfs.service
    85ms remount-rootfs.service
    65ms home.mount
    36ms dev-sda2.swap
    13ms nfs.service

Not to mention „systemd-analyze plot > bootchart.svg“. Anyway. We’ll start by something else. Currently I haven’t migrated completly to systemd. There are still some old sysv-init.d scripts in use:

wdp@localhost ~ $ systemctl | grep SYSV
ConsoleKit.service        loaded active running       SYSV: The ConsoleKit maintains a list of sessions
hostname.service          loaded active exited        SYSV: Sets your hostname
nfs.service               loaded active exited        SYSV: NFS is a popular protocol for file sharing across TCP/IP networks. This service provides NFS server functionality, which is configured via the /etc/exports file.

So before I do anything else, I should migrate that stuff. Right after doing so I should take a look at the „failed“ services:

wdp@localhost ~ $ systemctl --failed
systemd-...-clean.service loaded failed failed        Cleanup of Temporary Directories
systemd-...-setup.service loaded failed failed        Recreate Volatile Files and Directories

And sadly, that’s not even all:

[    2.943944] systemd[1]: No hostname configured.
[    2.945344] systemd[1]: Cannot open /etc/machine-id: No such file or directory
[    5.312286] systemd-fsck[1335]: Usage: fsck.ext4 [-panyrcdfvtDFV] [-b superblock] [-B blocksize]
[    5.312296] systemd-fsck[1335]: [-I inode_buffer_blocks] [-P process_inode_size]
[    5.312303] systemd-fsck[1335]: [-l|-L bad_blocks_file] [-C fd] [-j external_journal]
[    5.312309] systemd-fsck[1335]: [-E extended-options] device
[    5.312315] systemd-fsck[1335]: fsck failed with error code 16.
[    5.312322] systemd-fsck[1335]: Ignoring error.
[    5.320895] systemd-fsck[1344]: Emergency help:
[    5.323447] systemd-fsck[1344]: -p                   Automatic repair (no questions)
[    5.325841] systemd-fsck[1344]: -n                   Make no changes to the filesystem
[    5.328317] systemd-fsck[1344]: -y                   Assume "yes" to all questions
[    5.336325] systemd-fsck[1344]: -c                   Check for bad blocks and add them to the badblock list
[    5.338879] systemd-fsck[1344]: -f                   Force checking even if filesystem is marked clean
[    5.341557] systemd-fsck[1344]: -v                   Be verbose
[    5.345369] systemd-fsck[1344]: -b superblock        Use alternative superblock
[    5.451610] systemd-fsck[1344]: -B blocksize         Force blocksize when looking for superblock
[    5.454516] systemd-fsck[1344]: -j external_journal  Set location of the external journal
[    5.457696] systemd-fsck[1344]: -l bad_blocks_file   Add to badblocks list
[    5.460924] systemd-fsck[1344]: -L bad_blocks_file   Set badblocks list
[    6.145793] systemd-vconsole-setup[1340]: /bin/setfont failed with error code 1.

I guess you agree, that it doesn’t make much sense to tune something, which throws errors and warnings 🙂 Let’s go through it step by step:

[ 2.943944] systemd[1]: No hostname configured.

Notice: Fix committed to lunar.

root@localhost / # echo "lunar" > /etc/hostname
root@localhost / # 

[ 2.945344] systemd[1]: Cannot open /etc/machine-id: No such file or directory

Notice: Fix committed to Lunar

root@localhost / # systemd-machine-id-setup 
Initializing machine ID from D-Bus machine ID.
root@localhost / # cat /etc/machine-id 
e5050307ea0f8ac1cf280a8900000010

the ext4 fsck error

This one is somewhat special, basically you can find out by looking at the source of systemd (src/fsck.c) that systemd-fsck uses a few parameters, for example: -T and -M. Both aren’t available in fsck.ext4. That’s causing the error. So there are two options:

  • patch systemd
  • write a wrapper for fsck.ext4 (e2fsprogs)

systemd-…-clean.service loaded failed failed Cleanup of Temporary Directories

Notice: Fix committed to Lunar

This happens because the systemd devs or the systemd dev wrongly assumed that there’s a „lock“-group on every linux-installation. At least that’s not the case for lunar. Simply changing group „lock“ to „root“ in:

/usr/lib/tmpfiles.d/legacy.conf

will do the job.

The SYSV-Stuff

I’m just re-installing systemd now and saying „no“ to sysv-compat because I’ve set the hostname and there’s no interesting sysv-init script anymore which needs to be started.

Let’s see what I get after rebooting:

wdp@lunar ~ $ systemd-analyze 
Startup finished in 3101ms (kernel) + 8333ms (userspace) = 11435ms

So, you can see, there are some fluctuations at the kernel time (between 2,9 and 3,1) however, in userspace we’ve got an time improvement of 0,9 (so, nearly a second). And, just to point that out again: This is WITH LOADED x11 and network.

Let’s continue to fix stuff:

[ 4.173874] systemd[1]: Cannot add dependency job for unit console-kit.service, ignoring: Unit console-kit.service failed to load: No such file or directory. See system logs and ’systemctl status console-kit.service‘ for details.

Notice: Fix committed to Lunar

This issue is caused by a missing option in our build script of console-kit. I already committed a fix for this.

[ 6.397777] systemd-vconsole-setup[1338]: /bin/setfont failed with error code 1.

This one can be ignored. In my case (and if you see it, most likely in your case, too) this is caused by me using ati-kms, so setfont can’t change the fonts.

Tuning of services

Alright, in the first part of this article I optimized the kernel to waste less time. In the second part I was fixing problems with our systemd integration in Lunar Linux and various services. The next part would be about tuning the services itself properly. Tho I’ll write a new article for that.

Last state btw:

wdp@lunar ~ $ systemd-analyze
Startup finished in 3085ms (kernel) + 7333ms (userspace) = 10418ms

i.e. boot with x11 and networking fully working, in 10 seconds.

No Comments

Post a Comment