Confused by the output of ‘du’ in Linux?

One of the questions on Super User that really hasn’t had the exposure it deserves is this one asked by Jason:

I am confused with the output of command ‘du’ in Linux.

Actually, it’s not so much a question and trying to understand some confusion, so let’s take a look at what he’s confused over and pull it all apart shall we?

So when he creates a file that is 1120 bytes in size and looks at it with ‘du‘, if he gets the ‘Human Readable’ form of the result, -h, it claims the file is about 4 times the size that it actually is. But, if he requests the number of bytes, -b, it shows the real size. What is going on here?

Well, this confusion all stems from two things:

1. A misunderstanding of the function of the du command, and
2. Certain facts about how a filesystem stores a file.

The two are so closely interrelated that we’re going to have to tackle them both at once by looking at what is actually going on when you store a file on disk.

As you may well know, before you use a new hard drive, you have to format the disk. This puts some special data in certain locations on the drive so the operating system can know where the data is on the drive. Sort of an index, if you like. Different operating systems and filesystems use different terminology for this data. In MS-DOS it is the FAT, or the File Access Table. In Linux and other Unix systems it’s often called the superblock. On the old Amiga systems it was referred to as the Filesystem Bitmap. Whatever the name the purpose is the same – to keep track of what data is where. For the sake of argument, let’s call this the superblock from here on.

Now, a hard drive is made up of a number of platters, round disks, the surface of which is used to store the data. This surface is split up into a number of tracks, or concentric rings. Each track is then subdivided into a number of sectors.

Typically a sector is 512 bytes in size. These sectors are traditionally what the superblock keeps track of. Why keep track of chunks of 512 bytes instead of individual bytes, you may well ask. It sounds a little inefficient, doesn’t it? Well, yes and no. It would be fantastic if you could track the allocation of each and every byte on the hard disk, but imagine the size of the superblock! It would take up the entire disk! So sectors were thought up as a good compromise. What it means is that a file when written to the disk will always take up at least one whole sector. If the file is less than 512 bytes, it will be allocated a whole sector of 512 bytes. If it’s 1120 bytes it will be allocated 3 whole sectors – 2 sectors are 1024 bytes, plus 96 allocated to the third sector, taking up in reality 1536 bytes.

But that doesn’t quite add up to what Jason was seeing, does it?

Well, sectors were all very well when hard drives were quite small, but as hard drives grew in size, so did the number of sectors on the drive increase. This meant that the superblock got bigger and bigger and more and more difficult to deal with. So a new concept was introduced – Clusters.

Clusters are groups of sectors – kind of like super-clusters. The number of sectors in a cluster is entirely down to the operating system and the filesystem, and can often be tuned when you actually format the hard drive. A typical size is 8 sectors to the cluster, and this is what Jason has on his filesystem. The superblock now keeps track of the clusters, instead of the individual sectors, making for a smaller, more manageable superblock size.

So a file of 1120 bytes is allocated one cluster, which is 8 sectors, which is 8 x 512 = 4096 bytes (or 4 kilobytes), exactly what Jason was seeing with du -h.

You see, the du command is supposed to show the amount of space the file is using on the disk, not the size of the file. du stands for Disk Usage. So in this respect the 4.0K that du -h is reporting is 100% correct.

The du -b command actually tells you the size of the file itself – the same that would be reported by the ls command. This is not really the original intended operation of the du command, but has been added by popular demand at some point in the past.