Blog Posts tagged ‘insufficient’

A customer got in touch with us recently saying they couldn’t upload files via FTP due to insufficient diskspace, but there was plenty of free space apparent when they logged in and checked. We don’t normally manage their server, but we said we’d take a look.

root@aria:~# touch /srv/www/newfile
touch: cannot touch `/srv/www/newfile': No space left on device

After logging in and taking a look around, the problem became apparent.

root@aria:~# df -h
Filesystem                Size  Used Avail Use% Mounted on
/dev/mapper/nayuki-root  1016M  536M  429M  56% /
/dev/mapper/nayuki-usr    4.0G  1.3G  2.6G  34% /usr
/dev/mapper/nayuki-var    4.0G  1.3G  2.6G  33% /var
/dev/mapper/nayuki-www     50G   41G  6.4G  87% /srv/www

There’s plenty of disk space…

root@aria:~# df -i
Filesystem                Inodes   IUsed   IFree IUse% Mounted on
/dev/mapper/nayuki-root    65536    7244   58292   12% /
/dev/mapper/nayuki-usr    262144   96830  165314   37% /usr
/dev/mapper/nayuki-var    262144   19680  242464    8% /var
/dev/mapper/nayuki-www   3276800 3276800       0  100% /srv/www

But no spare inodes!

Inodes are a filesystem structure used to store metadata about files (we use ext3, the default, so the following details may not necessarily apply to other filesystems). This of course takes up space on the disk, so inodes are preallocated when the filesystem is first created. The practical consequence is that you can’t create any more files once you run out of inodes. Not all filesystems work in the exact same way, so the system gives you a general error about being out of space.

This is usually correct – to be honest, I’ve never actually seen a filesystem run out of inodes before. It’s something of a nuisance that you can’t allocate extra inodes on an existing filesystem. The best short-term solution we could offer the customer was to remove or migrate some files.

If you use Linux’s Logical Volume Manager (LVM) subsystem, you can pull some trickery and grow an existing filesystem, but this doesn’t let you create an arbitrary number of spare inodes. If you’re going to have a massive number of files, the filesystem needs to be created from scratch with lots of empty inodes.

Chances are you’ll never run into this, but if you deal with systems that have a lot of small files (eg. mail servers, possibly public-facing web servers) it’s something to keep in mind when you set things up. Although it’s not something we’d run into before, we were able to identify the problem and provide a quick solution thanks to our extensive general knowledge of our systems.

Useful commands (ext2/3 filesystems on linux):

df -h
     diskspace utilisation on filesystems

df -i
     inode utilisation on filesystems

dumpe2fs -h /dev/nayuki/www
     get info on the filesystem (specify the path to the block device)