Automatic Snapshots in ZFS on Linux

Some months ago I did write a script which can be used to automatically create snapshots. I’ve became unhappy with it due to a few reasons and started a rewrite of it. So here we go again…

First of all, that script was not creating valuable snapshots. The most used example with ZFS Snapshots is probably the example which shows that ZFS snapshots only use space if there are changes:

root@janice:~# zfs snapshot storage/databases/innodb-data@test1
root@janice:~# zfs snapshot storage/databases/innodb-data@test2
root@janice:~# zfs snapshot storage/databases/innodb-data@test3
root@janice:~# zfs snapshot storage/databases/innodb-data@test4
root@janice:~# zfs list -d 1 -t snapshot storage/databases/innodb-data
NAME                                  USED  AVAIL  REFER  MOUNTPOINT
storage/databases/innodb-data@test1      0      -  5.54M  -
storage/databases/innodb-data@test2      0      -  5.54M  -
storage/databases/innodb-data@test3      0      -  5.54M  -
storage/databases/innodb-data@test4      0      -  5.54M  -

As you can see, creating snapshots is cheap. You can create many of them and only those which contain changes will eat up space. That is the problem I am dealing with: Why should I keep an endless list of empty snapshots representing a point in time? Actually, my use for a snapshot is to rollback to a point in time if I need specific data. Hence I do only need snapshots which actually contain changes. I call such snapshots valuable snapshots.

root@janice:~# zfs snapshot storage/databases/innodb-data@test5
root@janice:~# zfs snapshot storage/databases/innodb-data@test6
root@janice:~# zfs list -d 1 -t snapshot storage/databases/innodb-data
NAME                                  USED  AVAIL  REFER  MOUNTPOINT
storage/databases/innodb-data@test1      0      -  5.54M  -
storage/databases/innodb-data@test2      0      -  5.54M  -
storage/databases/innodb-data@test3      0      -  5.54M  -
storage/databases/innodb-data@test4      0      -  5.54M  -
storage/databases/innodb-data@test5    92K      -  5.54M  -
storage/databases/innodb-data@test6      0      -  5.54M  -

One out of 6 snapshots actually contains changes and all it does eat up is 92k. My last script did not care whether there are changes or not. I do only want those which contain them. There are two ways in Linux how you can achieve that (probably more, but these are the two I am aware of): 1. zfs diff 2. inotifywait. The latter can be used to monitor zfs filesystems (I did not try this, yet). However, I’d like to go with standard zfs utils and hence decide for the former. In Bash that could look like this:

root@janice:~# datasetHasChanges() {
>   local dataset="$1"
>   local snapshot=$(zfs list -H -d 1 -t snapshot -o name -S creation "$dataset" | awk 'NR==1');
>   local diff=$(zfs diff -H $snapshot $dataset);
>   if [ -z "$diff" ]; then
>     return 1
>   fi
>   return 0 
> }
root@janice:~# zfs snapshot storage/databases/myisam-data@initial
root@janice:~# if datasetHasChanges storage/databases/myisam-data; then echo "yup"; else echo "nope"; fi
nope

– restart of the database –

root@janice:~# if datasetHasChanges storage/databases/myisam-data; then echo "yup"; else echo "nope"; fi
yup

Using that function I can check if there have been changes before I create a snapshot. Problem one solved. The next problem with my old script is that it did not take account of the used space, easily filling up filesystems and causing all sorts of problems just by creating a snapshot. If you do use quotas, used size by snapshots is part of the used size of the filesystem. That means a filesystem with a quota of 5 GB and 2 GB of snapshots has just 3 GB free for use. My script should take care of that and should not create snapshots if there is less than 5% free. The following command can be used for that:

zfs get -Hp available -o name,value -t filesystem

-H removes the header, -p displays exact values (which are easier to parse). Part of the output:

storage/databases       3715041779712
storage/databases/innodb-data   3715041779712
storage/databases/innodb-logs   3715041779712
storage/databases/myisam-data   3715041779712
storage/images  106226286592

A simple method which could do the check is:

root@janice:~# datasetHasEnoughSpace() {
>   local dataset="$1"
>   local factor="$2"
>   local used=$(zfs get -Hp used -o value -t filesystem "$dataset")
>   local avail=$(zfs get -Hp available -o value -t filesystem "$dataset")
>   local limit=$(((($used+$avail)*$factor)/100))
>   if [[ $avail -gt $limit ]]; then
>     return 0
>   fi
>   return 1
> }
root@janice:~# if datasetHasEnoughSpace "storage" "5"; then echo "yup"; else echo "nope"; fi
yup
root@janice:~# if datasetHasEnoughSpace "storage" "95"; then echo "yup"; else echo "nope"; fi
yup
root@janice:~# if datasetHasEnoughSpace "storage" "99"; then echo "yup"; else echo "nope"; fi
nope

You might implement additional logic like increasing a quota by the amount of space used by a snapshot every time you create one. I’ve been thinking about that but I do not need such functionality; It might introduce other problems. Instead I’ll implement some sort of mail notification if that 95% are reached. Problem two solved.

The third problem is more like a nice-to-have rather than a problem. Actually the first script I wrote took a list of filesystems to exclude. The next one took a list of filesystems to include. Instead of that I’d like to use the com.sun:auto-snapshot property, because it is inherited by children datasets and allows better control of which datasets to in- and exclude:

root@janice:~# zfs set com.sun:auto-snapshot=true storage
root@janice:~# zfs get com.sun:auto-snapshot storage/databases 
NAME               PROPERTY               VALUE                  SOURCE
storage/databases  com.sun:auto-snapshot  true                   inherited from storage

Another problem solved and one remains. The last problem with my initial script is about the way I did create and destroy snapshots. Right after creation of 4 weekly snapshots it would have destroyed the oldest (1st) weekly snapshot once it created a new one (newest.. 5th). Because it does (or in regard to my first version might) contain valuable data I would just destroy data I might want to rollback to.

snapsMy new script should make use of the zfs rename functionality by renaming a snapshot to the upper period rather than destroying it. With period I do refer to yearly, monthly, weekly, daily, hourly and recently. A recently snapshot should go to hourly, a hourly snapshot should go to daily and so on. I had some trouble getting this logic to work. I’ve made a very simple flow chart for that which you can see on the right.

The order in which I iterate through the periods is important here. Starting with yearly, through monthly, weekly, daily, hourly and finally ending with recently. In every period I first check if there is a snapshot in the lower period and if so take the oldest snapshot of that. Means when creating a yearly snapshot I do check for the oldest monthly snapshot and take that. If there’s none I’d just skip yearly and do the same in monthly. If there’s none I’d just skip monthly and do the same in weekly and so on. Until I reach recently – in which I’ll always create a snapshot.

Now if I’d throw away a snapshot once a specific limit is reached I don’t throw away useful data because the upper period has a snapshot already.

I tend to reinvent the wheel due to learning purposes. That means if you are here because you’re looking for a mature solution to create automatic snapshots – You might want to take a look at two other scripts which might do a better (or similar) job than my script does. There is zfs-auto-snapshot on the one hand and zfs-snap-manager on the other – Both worth a look.

Anyway, a first version of my script can be found here: auto-snapshots.tar.gz. Basically remove or add periods as you like, make sure the most frequent period does not contain _next and all others should contain _next. _next is used to determine where to check for the snapshot which should be renamed.

The first few runs produce the following snapshots

NAME                                                     USED  AVAIL  REFER  MOUNTPOINT
storage/netboot@recently-2015-12-20113014                   0      -    96K  -
storage/netboot/kodi@monthly-December                    100M      -  3.82G  -
storage/netboot/kodi@weekly-51                           448K      -  3.82G  -
storage/netboot/kodi@daily-Sunday                       7.04M      -  3.98G  -
storage/netboot/kodi@hourly-12                           816K      -  3.90G  -
storage/netboot/kodi@recently-2015-12-20121552           372K      -  4.02G  -

The script will first fill up the periods with one snapshot (one recently, one hourly, one daily and so on) by renaming and create a recently snapshot. Seems to do exactly what I want it to do.

Update

thats how it does look like after a few hours

NAME                                                     USED  AVAIL  REFER  MOUNTPOINT
storage/netboot@recently-2015-12-20113014                   0      -    96K  -
storage/netboot/kodi@yearly-2015                         100M      -  3.82G  -
storage/netboot/kodi@monthly-December                    448K      -  3.82G  -
storage/netboot/kodi@weekly-51                          7.04M      -  3.98G  -
storage/netboot/kodi@weekly-52                           848K      -  3.90G  -
storage/netboot/kodi@daily-Monday                        224K      -  4.02G  -
storage/netboot/kodi@hourly-19                           208K      -  4.02G  -
storage/netboot/kodi@hourly-20                           244K      -  4.02G  -
storage/netboot/kodi@hourly-21                           424K      -  4.02G  -
storage/netboot/kodi@hourly-22                           104K      -  4.02G  -
storage/netboot/kodi@hourly-23                           128K      -  4.02G  -
storage/netboot/kodi@hourly-00                           368K      -  4.02G  -
storage/netboot/kodi@recently-2015-12-20233037           288K      -  4.02G  -
storage/netboot/kodi@recently-2015-12-20234538           296K      -  4.02G  -
storage/netboot/kodi@recently-2015-12-21000041           304K      -  4.02G  -
storage/netboot/kodi@recently-2015-12-21001537           288K      -  4.02G  -

The script is just added as cronjob:

*/15 * * * * /root/snapshot.sh

which just runs all 15 minutes.

No Comments

Post a Comment