Client Area

Debian Server Disk Space Full: Causes and Best Practices for Diagnosis & Resolution

Running out of disk space on a Debian server (Debian 12 “bookworm” or Debian 13 “trixie” in 2026) is one of the most common operational issues. A full filesystem triggers cascading failures: services fail to write logs or temporary files, updates stall, new connections drop, and in extreme cases the system becomes unresponsive or unbootable.

The problem is rarely the base OS itself consuming excessive space — a minimal Debian server install occupies ~1–3 GB. Instead, space exhaustion stems from runtime accumulation, misconfiguration, or unbounded growth in specific directories.

Primary Theoretical Causes

  1. Journald Persistent Logs (/var/log/journal) Since Debian 10+, systemd-journald defaults to persistent storage. On busy servers (high traffic web, containers, databases, monitoring agents), journals grow without bound if retention limits are not set.
    • Theoretical issue: unbounded append-only binary logs → linear consumption over time.
    • Symptom: /var/log/journal grows to tens or hundreds of GB over months/years.
  2. Traditional Log Files (/var/log/*.log, kern.log, syslog) If rsyslog is installed and misconfigured (or logrotate fails/crashes), active logs or rotated .gz files balloon. Kernel flood (buggy driver, hardware fault) or verbose application logging accelerates this.
  3. Package Cache & Archives (/var/cache/apt/archives) Downloaded .deb packages accumulate unless cleaned. On servers doing frequent apt upgrade or building containers, this can reach several GB quickly.
  4. Temporary Files (/tmp, /var/tmp) Applications, builds, Docker/Podman containers, or crashed processes leave behind large temporary files. /tmp is often tmpfs (RAM-backed), but /var/tmp is disk-backed and persistent.
  5. Deleted-but-Still-Open Files A process holds an open file descriptor to a deleted file → space is not reclaimed until the process exits or restarts.
    • Classic symptom: df shows full, but du -sh /* reports far less usage.
  6. Reserved Blocks on ext4/ext3 Filesystems By default, ext4 reserves 5% of space for root only (prevents total lockout when non-root fills the disk). When usage hits ~95%, non-root writes fail even if df shows a few percent free.
  7. Other Common Culprits
    • Docker/Podman overlay2 storage (/var/lib/docker) if not pruned.
    • Database data/wal files (/var/lib/postgresql, /var/lib/mysql) with no vacuum/autovacuum tuning.
    • User caches in /home or /root (e.g., .cache, pip/npm/go caches).
    • Core dumps (ulimit -c unlimited + crash loop).
    • Mount-point hiding (data written under /var before mounting a separate /var filesystem).

Step-by-Step Diagnosis Approach

  1. Confirm which filesystem is full Use human-readable output to see real usage and inodes:
    • df -hT (filesystem type matters — ext4 vs xfs vs btrfs behave differently)
    • df -i (inode exhaustion can mimic space exhaustion)
  2. Find large directories/files (non-root safe)
    • sudo du -hxd1 / | sort -hr → top-level breakdown
    • sudo ncdu / (interactive ncurses tool — install with apt install ncdu) — fastest way to navigate and spot offenders
  3. Check for deleted-but-open files
    • sudo lsof +L1 or sudo lsof / | grep deleted
    • Or: sudo find /proc/*/fd -ls 2>/dev/null | grep ‘(deleted)’ → Restart or kill the offending process to reclaim space.
  4. Check reserved blocks (ext4)
    • sudo tune2fs -l /dev/sdX | grep ‘Reserved block count’
    • Reduce if safe: sudo tune2fs -m 1 /dev/sdX (1% reserve, or 0 on large disks)

Targeted Fixes by Common Cause

  • Journald logs Limit size/retention: edit /etc/systemd/journald.conf
    text
    SystemMaxUse=500M
SystemKeepFree=2G
MaxRetentionSec=1month

    Then sudo systemctl restart systemd-journald One-time vacuum: sudo journalctl –vacuum-size=500M or –vacuum-time=2weeks

  • APT cachesudo apt clean (removes all .deb) sudo apt autoclean (removes obsolete .deb) sudo apt autoremove –purge
  • Log files Force rotation & cleanup: sudo logrotate -f /etc/logrotate.conf Truncate active logs if safe: sudo truncate -s 0 /var/log/kern.log Remove old rotated logs: sudo find /var/log -type f -name ‘*.gz’ -mtime +30 -delete
  • Temporary filessudo rm -rf /tmp/* /var/tmp/* (safe on reboot; use caution on live systems)
  • Docker / containers (if applicable) docker system prune -a –volumes Or podman equivalent.

Prevention Strategies

  • Monitor disk usage proactively (Netdata, Prometheus/node-exporter alerts on >85% usage).
  • Set journald limits by default on new servers.
  • Schedule weekly cleanup cron jobs for caches/logs.
  • Use separate partitions for /var, /var/log, /var/lib/docker if possible.
  • Consider tmpfs for /tmp on low-RAM servers (but watch memory pressure).
  • Regularly run apt autoremove && apt clean.

A full disk is almost always fixable without reinstall — the key is rapid diagnosis with tools like ncdu, lsof +L1, and targeted cleanup. Start with df -h and ncdu / — they reveal 90% of cases immediately.

Leave a Reply