Inode shortage reaches critical levels
A customer got in touch with us recently saying they couldn’t upload files via FTP due to insufficient diskspace, but there was plenty of free space apparent when they logged in and checked. We don’t normally manage their server, but we said we’d take a look.
[email protected]:~# touch /srv/www/newfile touch: cannot touch `/srv/www/newfile': No space left on device
After logging in and taking a look around, the problem became apparent.
[email protected]:~# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/nayuki-root 1016M 536M 429M 56% / /dev/mapper/nayuki-usr 4.0G 1.3G 2.6G 34% /usr /dev/mapper/nayuki-var 4.0G 1.3G 2.6G 33% /var /dev/mapper/nayuki-www 50G 41G 6.4G 87% /srv/www
There’s plenty of disk space…
[email protected]:~# df -i Filesystem Inodes IUsed IFree IUse% Mounted on /dev/mapper/nayuki-root 65536 7244 58292 12% / /dev/mapper/nayuki-usr 262144 96830 165314 37% /usr /dev/mapper/nayuki-var 262144 19680 242464 8% /var /dev/mapper/nayuki-www 3276800 3276800 0 100% /srv/www
But no spare inodes!
Inodes are a filesystem structure used to store metadata about files (we use ext3, the default, so the following details may not necessarily apply to other filesystems). This of course takes up space on the disk, so inodes are preallocated when the filesystem is first created. The practical consequence is that you can’t create any more files once you run out of inodes. Not all filesystems work in the exact same way, so the system gives you a general error about being out of space.
This is usually correct – to be honest, I’ve never actually seen a filesystem run out of inodes before. It’s something of a nuisance that you can’t allocate extra inodes on an existing filesystem. The best short-term solution we could offer the customer was to remove or migrate some files.
If you use Linux’s Logical Volume Manager (LVM) subsystem, you can pull some trickery and grow an existing filesystem, but this doesn’t let you create an arbitrary number of spare inodes. If you’re going to have a massive number of files, the filesystem needs to be created from scratch with lots of empty inodes.
Chances are you’ll never run into this, but if you deal with systems that have a lot of small files (eg. mail servers, possibly public-facing web servers) it’s something to keep in mind when you set things up. Although it’s not something we’d run into before, we were able to identify the problem and provide a quick solution thanks to our extensive general knowledge of our systems.
Useful commands (ext2/3 filesystems on linux):
df -h diskspace utilisation on filesystems
df -i inode utilisation on filesystems
dumpe2fs -h /dev/nayuki/www get info on the filesystem (specify the path to the block device)