• AMD Athlon(tm) 64 X2 Dual Core Processor 3800+
• 3 GB Ram
• 3x WDC WD5000AADS-00S9B0 (500 GB) in SW Raid 5
• nVidia Corporation G98 [GeForce 8400 GS]
• Conexant Systems, Inc. CX23885 PCI Video and Audio Decoder (DVB-T Card)
• nVidia Corporation MCP61 High Definition Audio (snd-hda-intel)
• LITE-ON DVDRW LH-20A1P, ATAPI CD/DVD-ROM drive

Linux Boot Step by Step

To give you an idea how the boot-process works in Linux, here’s a little step by step overview about what happens when:

1. Boot-loader
2. Kernel
3. Init
4. Services

Before you start to tune the boot-time of your System you should set yourself some goals. Don’t set unrealistic goals. My goals are:

• Kernel max 3 seconds
• Init-Scripts max 7 seconds

(a bit later you will notice that i switch from sysvinit to runit – and thus the goals change to: one-time-scripts: max 3 seconds, rest max 4 seconds) – This way i should achieve a boot-time of 10 seconds.


Another note: I’ll write about the disk-throughput in my examples below, however i don’t think this is a helpful value as this might change from boot-to-boot.


Current – Sysvinit, default kernel

Alright, let’s take a look at the current boot time. As you can see in this graphic my current boot is 47 seconds. According to the graphic we have 9 seconds for the Kernel till init starts and 38 seconds till the system is “ready to use”. You can see easily that every service is started one-by-one so if you have 10 services, and each is taking 2 seconds, we’re talking about 2*10 seconds. Now take a look at the top of the graphic, you’ll see that the disk throughput (31 mb/s) is not very good and the system is more or less “bored” while booting (which simply means, the system could do a lot more in the same time)

This is a big problem with SysvInit – But gladly there are sysvinit replacements around. For example runit, upstart and systemd. I personally dislike systemd and upstart – Upstart because of its dependencies to python and dbus. Systemd because it seems to be really buggy at the moment and the on-demand feature is imho something which shouldn’t go into an init-system (and let’s be serious – re-inventing xinetd/inetd makes which sense?) plus the fact that systemd needs cgroupthingy and thus won’t run on non-linux-platforms.


sysvinit to runit – minimal migration

After switching sysvinit to runit, and doing minimal migration of the services (runit is sysv compatible – so i just wrote all the init scripts except for a few into /etc/runit/1) i got this graphic, on which you can see the boot-time is at 36 seconds (11 seconds won) and the throughput is at 38 MB/s (7 MB/s won). The kernel takes only 7 seconds anymore (without doing anything to it – so no idea why this happened).

runit – fully migrated

After migrating everything possible (talking about the services from sysvinit scripts to runit-run-scripts) i got a screen with a boot time of 27 seconds (won another 9 seconds) and a throughput of 91 MB/s (which is near to the limit of my hard discs and an improvement of another 53 MB/s). Kernel time is still around 7 seconds.

So – the only thing we did was: Replacing SysVinit with Runit and migrating all services. Just by doing so we boot in 27 seconds of 47 (20 seconds less), and got a throughput of 91 MB/s instead of 31 (60 MB/s more). Of course, thats not all, and i still didn’t reached the goal of a 10 seconds boot. It’s time to take a look at the kernel.


modifying the kernel – remove not needed stuff

In this graphic you will see that i have a boot-time of 23 seconds (we won around 4 seconds) and a throughput of 68 MB/s (we lost around 23 MB/s). All i did was removing not needed modules and not needed stuff from the default Lunar Linux kernel, enabling preempt (low latency desktop). The kernel-time went down from ca 7 seconds to 5 seconds – so we won 2 seconds. However, the goal was 3 seconds – So i still need to do something with the Kernel. The easiest thing to have the Kernel at a 3 seconds boot is to just make everything which is not needed for a boot modularized. For example your USB-mouse and keyboard, your CD-ROM drive, your DVB-t card and similar stuff. That stuff doesn’t need to be available while booting the kernel, so make it a module…

make everything not important modular

…and i get this screen – which shows 24 seconds boot time (so we lost 1 second) and a throughput of 56mb/s (lost 12 MB/s) – However: The kernel-time is only around 3,2 up to 3,3 seconds instead of 5 seconds – So I’d say first goal achieved – Kernel needs only around 3 seconds. I’m sure it’s possible to boot the kernel in 2 seconds – do dmesg | less and enable kernel time display so that you can see what is “hanging” where – you can tune the time then by making things modular or by disabling them.

modify init/run scripts

As you can see in the latest graphic the time from running rcS to start the services in parallel is about 13 seconds long. As you might remember the goal was to have the one-time-scripts done/started within 3 seconds. So we’ll tune this now by modifying rcS.

Due to these changes the boot-time goes down from 24 seconds to 17 seconds – so we made it take 7 seconds less. And only 7 seconds to our “goal”. The time of the one-time-scripts went down to ca 5 seconds. So we need to get 2 more seconds out here. This means hacking the udev and mount script. Remember, we went down from 47 to 17 seconds already – and we didn’t touch mount, udev, x11 – So you’re booting Linux with x11 plus networking in 17 seconds.

hacking the udev and mount init script

For the mount script, which takes about 5 seconds, do the following: comment wait_some everywhere (except for the function at the start). Then remove the 2.4 kernel compatibility stuff and finally comment out LVM stuff (if you dont need it – i don’t).

Wow – Booting in 15 seconds.


To be continued… :)

Other Links

• Lunar Linux
• Boot Linux faster

alia:~# killall -9 sshd
Connection to XX.XXX.XX.X closed by remote host.
Connection to XX.XXX.XX.X closed.
wdp@localhost:~$ ssh XX.XXX.XX.X -l wdp -C
wdp@XX.XXX.XX.X's password:

So – Yes, a service is automatically restarted if it crashes. Other nice things are service-dependecies as shown here: http://smarden.org/runit/dependencies.html. For more run-scripts check this url: http://smarden.org/runit/runscripts.html.

By the way, here’s a pstree (yeah im not using pro-ftpd and ispcp_daemon with runit yet)

runit-+-ispcp_daemon
      |-proftpd
      `-runsvdir-+-runsv---xinetd
                 |-runsv---mysqld---18*[{mysqld}]
                 |-runsv-+-master-+-anvil
                 |       |        |-bounce
                 |       |        |-pickup
                 |       |        |-qmgr
                 |       |        `-smtp
                 |       `-svlogd
                 |-runsv---lard
                 |-runsv---cron
                 |-runsv-+-logger
                 |       `-sshd
                 |-runsv---dovecot-+-2*[dovecot-auth]
                 |                 |-9*[imap]
                 |                 |-3*[imap-login]
                 |                 `-3*[pop3-login]
                 |-runsv---nscd---8*[{nscd}]
                 |-runsv---named---6*[{named}]
                 |-runsv---apache2-+-apache2
                 |                 |-apache2---18*[php5-cgi---2*[php5-cgi]]
                 |                 |-apache2---11*[{apache2}]
                 |                 `-2*[ispcp-apache-lo]
                 |-runsv---postgrey
                 `-runsv---policyd-weight

• Lunar Linux
  • acpid
    run:

    #!/bin/bash
    
    # Enable IBM specific hotkeys by default
    if [ -e /proc/acpi/ibm/hotkey ]; then
            echo enable,0xffff > /proc/acpi/ibm/hotkey
    fi
    
    # Start acpid in foreground
    exec /usr/sbin/acpid -f
    

  • agetty
    run:

    #!/bin/sh
    exec /sbin/agetty 38400 tty1 linux
    

    finish:

    #!/bin/sh
    exec utmpset -w tty1
    

  • ConsoleKit
    run:

    #!/bin/sh
    exec /usr/sbin/console-kit-daemon --no-daemon
    

  • hald
    run:

    #!/bin/bash
    
    # run hald
    exec /usr/sbin/hald --daemon=no --use-syslog &> /dev/null
    

  • messagebus
    run:

    #!/bin/bash
    
    if [ -x /usr/bin/dbus-uuidgen ] ; then
            /usr/bin/dbus-uuidgen --ensure
    fi
    
    exec dbus-daemon --system --nofork &> /dev/null
    

  • portmap
    run:

    #!/bin/bash
    
    # run portmap
    exec /sbin/portmap -f -l
    

  • smartd
    run:

    #!/bin/sh
    exec /usr/sbin/smartd -n
    

  • sshd
    run:

    #!/bin/sh
    exec 2>&1
    exec /usr/sbin/sshd -D -e
    

  • udevd
    run:

    #!/bin/bash
    
    rm -rf /dev/.udevdb
    export ACTION=add
    exec /sbin/udevd
    /sbin/udevadm trigger
    /sbin/udevadm settle
    

  • wpa_supplicant
    run:

    #!/bin/bash
    
    # Start only if wlan0 is there, if not
    # wait 5 seconds and try again, still
    # not there, disable this service for now
    if /sbin/ifconfig | grep wlan0 ; then
            exec /usr/sbin/wpa_supplicant -i wlan0 -c /etc/wpa_supplicant.conf
    else
            # lets wait 5 seconds, maybe the device is coming up then
            sleep 5
            if /sbin/ifconfig | grep wlan0 ; then
                    exec /usr/sbin/wpa_supplicant -i wlan0 -c /etc/wpa_supplicant.conf
            else
                    # still not there, probably disabled. let's
                    # disable this service then.
                    /usr/local/bin/sv stop wpa_supplicant
            fi
    fi
    

  • xinetd
    run:

    #!/bin/bash
    
    # start xinetd
    exec /usr/sbin/xinetd -dontfork
    

• Debian (Lenny)
  • apache2
    run:

    #!/bin/sh
    exec 2>&1
    exec chpst -P apache2 -DNO_DETACH -k start -DSSL -DFASTCGI -f /etc/apache2/apache2.conf
    

  • cron
    run:

    #!/bin/sh
    exec 2>&1
    exec cron -f -l
    

  • dovecot
    run:

    #!/bin/sh
    exec 2>&1
    # my dovecot setup needs mysql
    svwaitup 3 /service/mysqld || exit 1
    # start dovecot in foreground
    exec /usr/sbin/dovecot -F
    

  • lard
    run:

    #!/bin/sh
    exec 2>&1
    exec lard -n
    

  • mysqld
    run:

    #!/bin/sh
    exec 2>&1
    exec /usr/sbin/mysqld                   \
            --basedir=/usr                  \
            --datadir=/var/lib/mysql        \
            --user=mysql                    \
            --pid-file=/var/run/mysqld/mysqld.pid \
            --skip-external-locking \
            --port=3306                     \
            --socket=/var/run/mysqld/mysqld.sock
    

  • named (bind9)
    run:

    #!/bin/sh
    exec 2>&1
    exec /usr/sbin/named -g -u bind 2>&1
    

  • nscd
    run:

    #!/bin/sh
    secure=""
    for table in passwd group
            do
                    if egrep '^'$table':.*nisplus' /etc/nsswitch.conf >/dev/null
                    then
                            nscd_nischeck $table || secure="$secure -S $table,yes"
                    fi
            done
    exec nscd -d -- $secure
    

  • policyd-weight
    run:

    #!/bin/sh
    exec 2>&1
    # policyd-weight wants postfix :)
    svwaitup 3 /service/postfix || exit 1
    exec /usr/sbin/policyd-weight -d start 2>&1
    

  • postfix
    run:

    #!/bin/sh
    exec 2>&1
    # my postfix setup needs mysql
    svwaitup 3 /service/mysqld || exit 1
    
    daemon_directory=/usr/lib/postfix               \
    command_directory=/usr/sbin                     \
    config_directory=/etc/postfix                   \
    queue_directory=/var/spool/postfix              \
    mail_owner=postfix                              \
    setgid_group=postdrop                           \
      /etc/postfix/postfix-script check || exit 1
    
    exec /usr/lib/postfix/master 2>&1
    

  • postgrey
    run:

    #!/bin/sh
    exec 2>&1
    # let's wait for postfix
    svwaitup 3 /service/postfix || exit 1
    
    exec /usr/sbin/postgrey --pidfile=/var/run/postgrey.pid --inet=127.0.0.1:60000 2>&1
    

  • sshd
    run:

    #!/bin/sh
    exec 2>&1
    exec /usr/sbin/sshd -D -e
    

ich bin gerade über etwas interessantes gestolpert (und werde hier ein bisschen zitieren, ich hoffe das ist in Ordnung und erlaubt) – Im Artikel “Der E-Postbrief: Post Scriptum” auf http://gutjahr.biz/blog/2010/07/post-scriptum/ zeigt der Author einen Brief der Post, in welchem es mir um folgende Auszug geht:

Zitat:

Die Inhalte der Nachrichten werden verschlüsselt abgelegt. Es ist nicht möglich – auch nicht für die Systemadministratoren – die Inhalte einzusehen. Das gesamte technische und organisatorische Sicherheitssystem ist so aufgebaut, dass die Vertraulichkeit der Nachrichten gewährleistet ist.

Quelle: http://gutjahr.biz/blog/2010/07/post-scriptum/

.

Hier musste ich etwas die Stirn runzeln. Diese Aussage “Es ist nicht möglich” brachte mich zu der Idee, mal zu überlegen wie man etwas solches realisieren könnte. Ich komme zu dem Schluß, mir wäre es nicht möglich. Eine eMail die ankommt oder ein Text der in einer Art Webmail gespeichert wird, könnte verschlüsselt auf einer Festplatte gespeichert werden – Die Daten die zum Entschlüsseln notwendig sind, müssen aber auch irgendwo gespeichert werden. Jeder der Zugriff auf diese Daten hat (Sei es bei einem Eingriff, oder sei es ein Systemadministrator), kann auch die eMails entschlüsseln und somit die Inhalte lesen. Wenn ich ganz viele Mitarbeiter habe, könnte ich vielleicht sagen das es Systemadministratoren gibt, die Zugriff auf die Entschlüsselungsdaten haben, aber keinen auf die Mails und umgekehrt. Aber das würde den administrativen Aufwand erheblich erhöhen und ich kann mir nicht vorstellen das es so in einem großen Unternehmen, gelöst werden würde (Ich kann mich hier aber irren). Wie auch immer – Spätestens wenn eine solche eMail ausgedruckt wird, muss sie entschlüsselt werden.

Weiter: Dank des Mediums Internet, ist es möglich, Verbindungen zu “sniffen” das heisst, ein Systemadministrator der die Verbindung überwacht, und jeder der halbwegs fit im Internet ist und eine Verbindung mitsnifft, könnte an die Inhalte der Mail kommen (sofern es nicht über Techniken wie SSL läuft).

Weiter: Würde ich eine solche Lösung bauen, und Postfix nutzen, würde ich eingehende eMails durch einen Filter laufen lassen, so wie amavisd, postgrey und ähnliches – Nur das dieser Filter die eMails vor der Zustellung an den naechsten Service (dovecot/lokale Zustellung (also Speicherung auf der Festplatte)), mit GPG verschlüsseln würde – Jeder Systemadministrator der dann in den Mailqueue schaut, kann an die Inhalte von diesen eMails kommen, bevor sie verschlüsselt werden. Mir ist kein Weg bekannt (es sei denn man nutzt ein eigens programmiertes Interface ohne Umweg über einen Mailserver – Vielleicht fehlt mir hier aber auch das Wissen um andere Mailserver, ich kenn und kann ja nur Postfix – Vielleicht könnte man sich extra dafür auch einen eigenen programmieren) um sowas direkt verschlüsselt zu speichern.

Meiner persönlichen (eigenen freien) Meinung nach, vermittelt eine solche Aussage (Zitat: “Es ist nicht möglich – auch nicht für die Systemadministratoren – die Inhalte einzusehen”) ein Sicherheitsgefühl, wo keines sein sollte – Dafür sind die Faktoren einfach zu unbekannt. Aber das ist wie gesagt, nur meine eigene Meinung.

Hmhm. Ich muss sagen, ich find’s trotzdem interessant und überlege mir da nen Account anzulegen bzw eine “Reservierung” so das ich das mal antesten kann. Jedoch, würde ich bei Dingen die nicht jeder lesen soll, immernoch auf den alten Postweg zurückgreifen, oder lieber direkt meine eMail per GPG Verschlüsselung an den Empfänger senden (Clientseitige Verschlüsselung ist das einzig sichere im Netz, meiner persönlichen Meinung nach.)

So long,
Jean

http://www.sysadminday.com/

Example

I tested it in Firefox 3 and Internet Explorer 8. Might be that it won’t run in every browser correctly. But thats a usual problem – By the way, in IE you might have a scrollbar cause of the height. There’s a fix for it but i don’t remember that one. That was setting a negative margin to the height or something. A shame that i forgot it. However, hopefully this snippet is helpful to someone :)

Side note: Adding filters might improve the look of your video, but it’s technically not improving the video, it’s making the video worse, as due to the filters, some information are lost – So this might not be what you want)

Repairing

I think there are a few ways to “repair” a video, however – It’s not possible to repair everything. Sometimes movies are shown two or three times on TV – If you store all two or three of them, you might average the 3 videos. So for example: Make sure all 2 or 3 (the more the better) videos start at the same, equal frame. Make sure you removed the advertisement, because this might change from video to video – The 3 videos needs to be equal in frames and length (that’s really important). Now you run all 2 or 3 videos through mplayer with -vo md5 and store those information in separate files – Now with a little bit bash experience you can compare those 3 files line by line to find frames which aren’t equal. If you have 3 videos this is quite cool, because you have just to look for the case, where 2 md5sums are equal and you might have a big chance that that is the “correct” frame – or it’s in both cases broken. You could also script some little gtk application which is showing you the frames so that you could compare
them manually. All frames which needs to be corrected, you’ll write into another file. Now the fun begins – Go through that new file (by another bash script) and export the specific frame of all 2 or 3 or the specific video as pictures. Now use ImageMagick or something else to average the picture. The next steps should be clear, you just need to use your corrected image now. This could be done if you export one whole video to images – and then you just replace the specific image with the newly created one, or you use avisplit, avimerge and or mencoder to split the video into two parts, insert the image as frame, merge the second part again. Just as you like.

Side note: If you have only two videos, and even with three or more videos, the problem with the “averaging” solution is, that you don’t know automatically which frame is the correct one and which is the broken one. So if you want to be sure, you need to check every frame manually. Which might be a lot of work – If you don’t and if the script detects a mismatching md5sum, you might average a correct frame with a broken frame. However, this is as far as i know the only thing you could do if you have artifacts or blocky parts in a video. By the way: I’m not using this. I was thinking about it but recording the same video three times is.. Well :)

Some other ways to repair such a video “stream” is the use of tools like ProjectX, which does a great job. In my humble opinion the most useful (at least for my videos) way of repairing such a video (or atleast trying to) is to run ProjectX onto the recorded mpeg. This will demux the video and try to repair a lot. After you did that, you’re using mplex to re-mux the video files. After doing that you use ffmpeg to convert your mpeg(mpeg1/2) to avi(mpeg4) by using -sameq and -vsync 1 – This way green boxes gets removed, if there are any. Another approach which i found in some DVB Wiki is using mencoder in the following way: mencoder -forceidx -lavdopts er=4 -vc ffmpeg12 -of mpeg -oac copy -ovc copy original_ps.mpeg -o cleaned_ps.mpeg So this might help you, also.

Side note: Do yourself a favour and don’t use this command – At least not without exactly knowing what it does. If you use it you’ll run into a few problems which have to do mostly with audio. Because: It seems to correct mpeg errors quite good, but if in your video someone is talking, and there’s some video glitch, though sound is okay, running this command will make the sound miss something. So – You might end up with someone talking in your video without finishing the sentence -
Why? I don’t know. Another problem i run into using this mencoder thingy was a/v sync. It seems in some rare cases using this will result in terribly a/v sync. So.. This command won’t get my suggestion :)

Let’s put it altogether, my way to repair/clean a dvb-recorded mpeg file is:

java -jar ProjectX.jar -demux file.mpg
mplex -f 8 -o muxed.mpg file.m2v file.mp2
ffmpeg -i muxed.mpg -vcodec mjpeg -sameq -vsync 1 -acodec copy repaired.avi

Finally you can use avidemux to remove advertisement, do that manually, select copy for both audio and video, select avi, if it asks you whether to use smart-copy say yes, and let the quantizer at the default value of 4 – Just save it as cutted.avi. You might also skip the ffmpeg part and do the avidemux part right after mplex – use mpg then and save as mpeg ps a+v.

That’s all. Let’s look at the file sizes. The original video got a file size of 2,6 GB (2675976). The muxed.mpg got a file size of 2,6 GB (2672448) – The repaired avi is a lot bigger (why i don’t know) 4.2 GB – And the cutted variant got 1,7 GB. This is still much, for one video in my humble opinion – And exactly this, brings us to the next topic.

Side note: To achieve the highest possible Quality, don’t do anything with your video, let it as it is. Probably just run it through projectx and mplex. And, if you want Quality, you shouldn’t care about file size. I’m a bit special, i want quality and a good file size, so i have to find a little compromise between both :)

Re-Encoding / Filtering

Right after a video is repaired/cleaned and cutted we can re-encode it to use another codec for video and audio and to find a nice home in some of the various container formats which are available. I don’t know why but i’m a big fan of h264, aac and mp4. Other people might like xvid, ogg and mkv (i don’t know whether that works together, never worked a lot with those) however – if you want to read more about the various codecs and containers, take a look at the mplayer documentation, there are a lot of codecs explained, more or less, i really like the mplayer documentation, probably a bit out of date, and sometimes not that easy to understand, but well – it’s good enough for what i need. However, when encoding a video we can do some
filtering also, like cropping black borders, blurring a bit, scaling, etc. So we have to talk about a bit more than just the encoding
process. Let’s start with the Aspect Ratio and Cropping.

Aspect Ratio

All movies I’m watching are recorded at a resolution of 720×576. One of them gets scaled to 768×576 (1.33:1) and thus is 4:3. Another one is scaled to 1024×576 (1.78:1) and thus is 16:9. Now there’s something special: a) i want all videos at 16:9. b) some movies have black borders around them, if we remove those, we have a totally different aspect ratio. c) after the cutting, sometimes the movie is switching the aspect ratio because of the advertisement which was in. I have to remove this.

So let’s work through this one by one. First of all, a way to “force” the aspect ratio and height and width:

-vf dsize=16/9,scale=720:576

Now let’s detect whether we need to crop black borders out. You can do this with the cropdetect filter, just run mplayer yourvideofile.avi -vf cropdetect. It will give you some crop line – scroll to a scene which is quite bright, so that it detects the black borders correctly. if needed adjust the value but make sure height and width are multiples of 16.

For example, my first movie is in 16:9 format already, and got 720×576. Cropdetect shows this line:

-vf crop=720:560:0:10

So it wants to remove 16px from the height – the 10 in the end just tells where the stuff is removed – 10 means, it’s picking 10 px from the top. so the last 6 px will be taken from the bottom. If we want it more in middle, you could use 8 instead of 10. So our whole line looks like:

-vf crop=720:560:0:8,dsize=16/9,scale=720:576

which will crop the video to 720x560px, set the aspect ratio to 16:9 and scale the video to 720×576 (the original) – as we remove only 16px this scaling isn’t noticeable i’d say.

Now let’s take a look at 4:3 videos. The following line, will do what we want:

-vf crop=704:464:10:32,dsize=16/9,scale=720:576

We’re in simply just removing the black borders from left and right (16px) and then we’re cropping the height to 464 – then we change the aspect ratio and scale it to 720×576 – voila’

Side note: Changing the Aspect Ratio doesn’t work loss less as far as i know without having extremely “stretched” videos – You might add black borders, but black borders are bad for efficiency and compression of videos, so this is not what you want. It’s better to leave the aspect ratio as is, instead of changing from 4:3 to 16:9, however the opposite, changing from 16:9 to 4:3 is quite easier and not so problematic.

Additional Filters

Now we’re coming to the interesting part :) At least i think this topic is quite interesting. However, i don’t want to explain them here as you can watch what they’re doing by taking a look at the man page of mplayer. So I’ll just give an overview about which settings i use:

-vf yadif=0,hqdn3d=4:3:4,pp=hb/vb/dr,harddup

yadif is a de-interlacer, doesn’t seem to hurt if pictures aren’t interlaced, hqdn3d is a high quality denoiser (so it’s blurring the
image), pp is a postprocessingfilter in this case using horizontal deblocking and vertical deblocking and deringing and harddup i just use so that a/v sync will be fine – The resulting video looks quite good. You could also give gradfun a try or use hqdn3d=6:4:6 though this is a bit much in some cases.

Side note: I got sometimes good results when using automatic brightness using al:f of the pp filters, however – in other cases the video looks too bright or too dark, and on my TV it’s looking differently than on my computer, thus don’t play around with that one. Another thing: I used to use pullup and softskip as the first two options, but somehow i got a feeling that this isn’t needed.. Anyway. Now you know which filters might be useful to you.

Another Side note about Re-Encoding: I used to download once a movie from a on-line recording site, because my local setup wasn’t working – The movie which i later downloaded had crappy quality. I was really disappointed about it – I already think i have bad quality if there are some glitches but that thing was just awful. While other people might
like that and are fine with that, I’m not. So what did i do? I run the whole movie through: mplayer movie.avi -sws 9 -vf
yadif=0,crop=640:352:0:4,dsize=16/9,scale=720:576,hqdn3d=4:3:4,pp=hb/vb/dr/al:f And voila’ the video looked useful. It’s a) deinterlacing if needed.. b) up scaling, c) denoising, d) deblocking and deringing, e) automatic brightness and contrast.

x264

x264 offers a lot of settings and comes with presets, the only settings i use are:

--preset slower --bframes 16 --ref 4 --crf 18

Just to give you an example, how i use this:

time x264 --preset medium --tune film --bframes 16 --ref 4 --crf 18 --output output.264 --no-fast-pskip video.y4m 2&gt:x264.log & mplayer cutted.avi -vo yuv4mpeg:file=video.y4m -benchmark -sws 10 -vf yadif=0,pp=ha/va/dr/al:f,crop=720:560:0:8,dsize=16/9,scale=720:576,hqdn3d=3:2:3,gradfun=1.2,harddup -nosound

Side note: You might want to tune those settings, it took 212 Minutes to encode a video using these settings (2x2GHz, 3GB Ram) i guess a lot of time is eaten due to the pp-filter (usually i use it without pp, depending on the video i use 3:2:3 or something else at hqdn3d and crf 16-20)

Additional Side note: It seems that sometimes the fifo isn’t working, remove the fifo using rm, mkfifo it again, wait 3 seconds and try again. I don’t know whats causing this. Take also a look at the various settings for x264, you might achieve better results than i. Some pages also state you have to add –fps25 and the width/height of the video, which I’m not doing in x264 as you can see, because it seems to work without that.

The Sound

I wrote an article about sound recently, how to turn stereo sound into surround sound (of course this is technically not “exactly” working) however, for this videos i won’t make myself the work, i just want to have the sound as is in AAC format. So I’ll dump it first from the latest created video file (cutted.avi), normalize it and then I’ll convert it to AAC using the following commands:

mplayer -ao pcm:fast -vc null -vo null -benchmark cutted.avi
normalize audiodump.wav
faac -q 100 -c 48000 --mpeg-vers 4 audiodump.wav

There you go. Let’s take a look at the file size:

original mp2: 114 MB
normalized pcm: 908 MB
our new aac: 78 MB

So… And – it sounds like the same, the quality seems to be equal.

Side note: Make sure your DVD player and/or equipment where you want to play the video later on, can do aac/x264. If your DVD player for example can’t you just wasted time :) You could also use ogg/vorbis, most players should be able to use that. To do so, do the same as above, except for the faac line, instead use: oggenc -R 48000 -q 10 -C 2 audiodump.ogg – Take also a look at the man page. :) Ah – And if you want to use ac3 instead of that, you might try: aften

Creating the Container

Now, as we good the sound and the video file processed, we should place them together into a little nice container. I’m using mp4 for that, someone else might want to use mkv or something else :)

MP4Box -fps 25 -add output.264 -add audiodump.aac movie.mp4

Yes! That’s all, that was it. :)

Side note: Right after packing it into mp4 you noticed that there’s a delay of 200ms in audio/video? Allright: You can export the stuff from the mp4, and add them right again with -delay. For example: MP4Box -raw 2 movie.mp4. You will want to use 1 and 2 to get audio and video out of the mp4. There might be more. Right after exporting it, do: MP4Box -add movie.h264 -add movie.aac -delay 2=200 test.mp4 and verify that test.mp4 works as expected.

File size of the mp4? 762 MB – Compared to the original 2,6 GB this is quite cool for a 90 Minutes video.

Another variant in shortform

The video from the online recorder. Cutted the video got a size of 170 MB. It’s 640x360px, divx (dx50, 25fps, 441.0 kbps) and mp3 with 128 kbit – So quite low quality. probably good for network streaming or some portable thingy.

I’m checking first using mplayer which filters might be useful to improve the quality and higher the resolution of that video. I came to the result that the following settings will do what i want:

yadif=0,crop=640:352:0:4,dsize=16/9,scale=720:576

this will deinterlace (doesnt hurt if not interlaced), scale it up to 720:576 and you will see some artifacts and blocky pictures. So the next step is to add some denoising, deblocking and deringing filters:

-vf yadif=0,crop=640:352:0:4,dsize=16/9,scale=720:576,hqdn3d=3:2:3,gradfun=2,pp=ha/va/dr

this does a quite good job. it will do the above explained things + denoise it, fix and dither banding artifacts, deblock and dering the video. Now we can do something with the brightness and contrast by adding al:f to the pp= list – This will make it a bit too dark for someones feeling but reduce additional “glitches” (let me use this word for blocky pixels, artifacts and similar stuff) and make the video look a bit more nicer compared to the original. Though the whole video looks a little bit unsharp now in some situations. So let’s add -sws 9 and change our -vf line to:

-sws 9 -vf yadif=0,crop=640:352:0:4,dsize=16/9,scale=720:576,hqdn3d=3:2:3,gradfun=2,pp=ha/va/dr/al:f,unsharp=l3x3:0.7

Of course.. The video doesn’t look like if it would have been recorded at that resolution but it looks similar and looks better than the original.

Side note: Remember, everytime you’re re-encoding, you’re losing more information. And everytime youre using filtering like gradfun, denoise, deblock, etc – you’re stripping additional information from the video or you’re modifying (thus loseing) them. So, even if the movie looks now a lot better, technically the movie is worse than the original.

So.. Step 1 – Create the fifo (1second?)

mkfifo video.y4m

Step 2 – Start the encoding (108 minutes)

time x264 --preset medium --tune film --bframes 16 --ref 4 --crf 20 --output output.264 video.y4m --no-fast-pskip 2>x264.log &  mplayer movie.avi -vo yuv4mpeg:file=video.y4m -benchmark -sws 10 -vf  yadif=0,pp=ha/va/dr/al:f,crop=640:352:0:4,dsize=16/9,scale=720:576,hqdn3d=3:2:3,gradfun=1.2,unsharp=l3x3:0.7,harddup -nosound

Step 3 – Dump the Audio ( a few seconds)

mplayer -ao pcm:fast -vc null -vo null -benchmark cutted.avi

Step 4 – Normalize the Audio ( a few seconds )

normalize audiodump.wav

Step 5 – Convert it to AAC (2 minutes, 56 seconds)

time faac -q 100 -c 48000 --mpeg-vers 4 audiodump.wav

Step 6 – Create the MP4 (a few seconds)

MP4Box -fps 25 -add output.264 -add audiodump.aac movie5.mp4

Result

431 MB file size. Quality is okay, some artifacts if there is text, sometimes a bit dark. – at least it’s looking better than the original, if i watch it in fullscreen on my widescreen tft.

Michael, a clanmate and friend told me that i could use an iPhone and Shazam to detect this song. So i exported some music from the movie, and let him try with his glory iPhone and shazam. The first result was something with japanese letters. The next one was “goodbyes by unknown” – However, the original title, it didn’t detect. Michael gave up with the comment “Well, probably too unknown” – Before this mess i told him “i bet this won’t work” – I won :)

However, i let him try three different songs, which are more known: Rise Against, Dancing for Rain || Pennywise, is anyone listening || Within Temptation, Our solemn Hour.

Shazam was able to detect all three. So this seems to be a quite good app for known songs which are in their database.

By the way, the song’s here: out of the movie “Professional, Story of a Killer” original title “One last dance” or the Song title “Broken Orange”.

I also switched back from my green layout (which i still like) to this lighty layout – I really really like this one, the only disadvantage is, if you show a lot of code like i, there’s not enough place. You got only like 330 px width for your code.. Anyhow. I also played a bit around with the Amazon Partner Net Widget’s and added the mp3 widget to my blog. Pity that it’s just a preview of 30 seconds, however it’s looking nice :)

In dem Buch “Göttin auf Zeit” von G. Haase-Hindenberg geht es um Amita, welche als Kind im Alter von 3 Jahren förmlich aus dem vertrauten Elternhaus gerissen wurde um als Gottheit für einen festgelegten Zeitraum, für die dortigen Menschen, zu leben.

Mir ist unklar ob das Buch fröhlich/glücklich oder traurig ist; Eigentlich ist es sehr traurig weil das Mädchen aus der gewohnten Umgebung gerissen wird, nur um ein paar Jahre später in dieselbe, dann total fremde Umgebung zurückgeschubst zu werden, quasi fallen gelassen wie eine heiße Kartoffel. Auf der anderen Seite erlebt das Kind auch viele schöne Dinge und manchmal kann man beim Lesen gar nicht anders als sich zu denken “Ach wie Süss” – Und alles in allem geht es der Kleinen ja auch nicht “so schlecht”.