Since 3 days i’m working on migrating 2 Servers into 1. This should have been an easy task, at least i hoped so. However, i noticed, that everytime i’m copying more than 41 GB to the new disc, i’m getting “Disk is full” errors. And i was really wondering, because “df -hT” shows that of 1 TB data only 41 GB are used. As the topic of this article might explain already: Yes. “df -i” shows that the inodes are full. This is due to a “bad” choice of me, when creating the filesystem. When creating an ext3 filesystem, you can tell it for what the filesystem will be used, for example news and large. This usage-parameter might be important because, if you have a lot of big files like on a storage, “large” might be a good choice. If you’re now trying to save all your mails and logfiles on this storage (usually small files) you’ll very soon run out of space, or to be correct: out of inodes. If you now use the opposite “news” instead of “large” you’ll have performance problems with big – very big files. The default value in my distribution seems to be a inode_ratio of 16384 (16 MB) by the way. “news” got 4 MB and “large” got an inode_ratio of 1 GB.

Nice that i was able to learn about that, however the problem still exists and you cannot change the inodes/inode_ratio on the fly while the system is running. So what to do? Happily i got two things: 1.) A Rescue System (netboot based) 2.) A Software Raid 5. So the plan is: Mark one disc of the raid array as faulty and remove it – Run the Raid array degraded. Prepare the “now” spare disc with a new filesystem, copy all files from the degraded raid array to the “spare disc”. re-make the filesystem of the degraded raid array, copy all files from the spare disc to the degraded raid array back. Voilá.

Got the same issues and the same luck of having a raid5 and rescuesystem or a spare disc? Searching for the commands? Here you go:

# set sda3 as faulty/fail in raid array md0
mdadm --manage /dev/md0 --fail /dev/sda3
# remove sda3 from the raid array md0
mdadm --manage /dev/md0 --remove /dev/sda3
# create new filesystem on spare disc
mkfs.ext /dev/sda3
# move into mnt folder, create two directories and mount them
cd /mnt
mkdir disk1
mkdir disk2
mount /dev/md0 disk1
mount /dev/sda3 disk2
# copy all files from the raid array to the spare disc
cd disk1
cp -rva * ../disk2/
# unmount the raid array
umount /dev/md0
# re-create the filesystem of the raid array with default settings
mkfs.ext3 /dev/md0
# OR: (this command might not be needed for you!)
# re-create the filesystem of the raid array with some special settings
# stride = chunk size of the raid array (256 here)
# stripe-width = (chunk size * Disks) - Parity
# -m 5% on 1 TB is quite much
mkfs.ext3 -E stride=256,stripe-width=512 -m 1 -O dir_index /dev/md0
# mount the new filesystem
cd ../
mount /dev/md0 disk1
# copy all files from the spare disc back to the raid array
cd disk2
cp -rva * ../disk1/

That’s it – Now reboot and hope the best. By the way. To re-add the “spare” disc into the raid array, you can use the following command, AFTER you verified that everything is running as expected:

mdadm --manage /dev/md0 --re-add /dev/sda3

You can follow the progress of resyncing by doing:

cat /proc/mdstat

Thanks to TheBonsai for helping!