df says the disk is full, du says it isn't. Who's right?

Both of them. df asks the filesystem how many blocks are allocated; du walks the directory tree and adds up what it can see. When those two disagree, something is holding space that no longer has a name. Here's how to find it.

1. Confirm the disagreement

$ clai compare what df reports for / with what du actually finds there→ df -h / && du -shx / 2>/dev/nullFilesystem      Size  Used Avail Use% Mounted on/dev/nvme1n1p2  915G  871G   44G  96% /348G	/

871 GB used, 348 GB found. That gap is the whole problem. -x keeps du from wandering into other mounted filesystems and inflating its own answer.

2. Check inodes before you chase bytes

$ clai check if the filesystem is out of inodes rather than space→ df -i /Filesystem        Inodes   IUsed      IFree IUse% Mounted on/dev/nvme1n1p2 124960768 3634368  121326400    3% /

If IUse% is at 100% while Use% isn't, you have millions of tiny files and no free inodes. That's a different problem — the disk has room, the filesystem has nowhere to record another file.

3. Find the deleted files still being held open

$ clai show files that were deleted but are still open by a process→ sudo lsof +L1COMMAND    PID USER   FD   TYPE DEVICE   SIZE/OFF NLINK  NODE NAMEpython3  23435 app     3u   REG  259,2 1000000000     0 41231 /var/log/app/debug.log (deleted)

NLINK 0 is the signature: no directory entry points at this file any more, so du cannot see it, but the blocks stay allocated until the last file descriptor closes. Usually it's a log someone rm'd without restarting the service that writes to it.

4. Reclaim the space

$ clai restart the service holding that deleted file→ sudo systemctl restart app.service

Restarting the process closes the descriptor and the kernel frees the blocks immediately. df drops the moment it happens — no unmount, no reboot.

5. If you can't restart it right now

$ clai truncate the deleted file that pid 23435 still holds on fd 3→ sudo truncate -s 0 /proc/23435/fd/3

Writing through /proc/<pid>/fd/<n> empties the file the process is still writing to, without killing it. The process keeps its descriptor and keeps logging; you get the space back now.

6. The other classic: a mount point hidden underneath

$ clai check whether something is mounted over a directory that still has files under it→ sudo mount --bind / /mnt/root && du -shx /mnt/root/var14G	/mnt/root/var

If files were written to /var/data before a disk was mounted there, they're still on the root filesystem — occupying space that nothing can see through the mount. A bind mount of / exposes them.

Gotchas

• Run du as root. As a normal user it silently skips directories it can't read, and you'll blame a phantom for what is really permissions.
• lsof +L1 needs privileges to see other users' descriptors. Without sudo you'll get a short, misleadingly clean list.
• Don't rm an active log again. That is what created this situation. Use truncate -s 0 on live log files, or configure logrotate with copytruncate.

Related questions

Why doesn't the space come back after rm? On Unix, unlinking removes the name, not the data. The blocks are freed when the last open descriptor closes — which for a running service means never, until it restarts.

du and df differ by a few percent, is that a problem? No. That's reserved blocks (5% for root by default on ext4) plus filesystem metadata. Only large gaps are worth investigating.

Can I find it without lsof? Yes: sudo ls -l /proc/*/fd 2>/dev/null | grep deleted shows the same thing through the process table.

See also

• "Who's eating my CPU right now?" — system monitoring in plain English
• Sweep old logs and backups with one sentence
• Delete files older than 30 days in Linux, safely

Stop looking up lsof flags at 3 AM — describe the problem and CliAI writes the command. Install it in one line.