The 2:14 AM Emergency
At 2:14 AM, my phone buzzed with a PagerDuty alert. The message was one every sysadmin dreads: CRITICAL: No space left on device (ENOSPC). Our primary shared development server had gone dark. I logged in via SSH and expected to find a bloated /var/log or a massive temporary file. Instead, df -h showed the /home partition was at 100% capacity.
Scanning the home directories with du -sh /home/* revealed the culprit. A junior developer had triggered a recursive backup script that dumped 450GB of raw sensor data into their home folder in under twenty minutes. Because we lacked resource governance, that single user killed the entire team’s workflow. The mail server stopped, cron jobs failed, and the system couldn’t even create a lock file to save a text document.
Why Filesystems Fail Without Limits
Linux is remarkably generous by default. It assumes users will share resources responsibly. However, in a production environment, relying on good behavior is a recipe for disaster. When a disk fills up, the kernel cannot write metadata, and applications crash immediately. It creates a total system paralysis.
The root cause of my late-night incident wasn’t just a buggy script; it was the absence of Disk Quotas. Without quotas, the filesystem treats every user as a part-owner of the total available space. To prevent this, you must enforce boundaries. You need a system that tells the kernel: “User A gets 10GB, User B gets 20GB, and the ‘Dev’ group gets 100GB total.”
Choosing the Right Tool
Several ways exist to handle storage limits. You could use LVM to create separate partitions for every user, but managing 500 individual volumes is a nightmare. XFS has its own built-in quota tools, which are excellent for high-performance setups. However, for standard ext4-based systems like Ubuntu 22.04 or Debian, the classic quota package remains the most reliable, battle-tested solution.
On a production server with 8GB of RAM and 2,000 active users, I found this kernel-level approach uses negligible CPU. It is far more efficient than running custom bash scripts to monitor disk usage every five minutes. Because it operates at the filesystem level, users cannot bypass limits by simply renaming files or hiding them in deep subdirectories.
Implementing Disk Quotas Step-by-Step
To ensure I never got that 2:14 AM call again, I deployed a standard quota system. You can set this up on your own server in about ten minutes.
Step 1: Install the Utilities
You need the userspace tools to communicate with the kernel. On Ubuntu or Debian, install them with this command:
sudo apt update
sudo apt install quota quotatool -yStep 2: Update the Filesystem Table
The kernel needs to know which partitions should track usage. Open /etc/fstab and find the partition you want to limit, such as /home.
# Original line in /etc/fstab:
UUID=xxxx-xxxx /home ext4 defaults 0 2
# Modified line for User and Group quotas:
UUID=xxxx-xxxx /home ext4 defaults,usrquota,grpquota 0 2Adding usrquota and grpquota tells the system to start tracking disk usage per user and per group during the next mount.
Step 3: Initialize the Quota Database
You don’t have to reboot the server. Remount the filesystem to apply the changes immediately:
sudo mount -o remount /homeNext, create the quota index files (aquota.user and aquota.group). The quotacheck command scans the disk and calculates current usage for every file on the system.
sudo quotacheck -cum /home-c: Create new quota files.-u: Check user-level quotas.-m: Perform the check without remounting as read-only.
Finally, activate the system:
sudo quotaon -v /homeStep 4: Setting Limits with edquota
The edquota command opens a temporary file in your default text editor, such as Vim or Nano, where you define specific limits.
sudo edquota -u developer_nameThe configuration will look like this:
Disk quotas for user developer_name (uid 1001):
Filesystem blocks soft hard inodes soft hard
/dev/sda2 409600 1048576 1258291 5 0 0Understanding these columns is critical:
- blocks: Current space used, measured in 1KB blocks.
- soft (limit): The warning threshold. Users can exceed this temporarily for a “grace period,” usually set to seven days.
- hard (limit): The absolute ceiling. The system will block all write operations once a user hits this number.
- inodes: The total number of files. This prevents users from creating millions of empty files that exhaust the filesystem’s metadata capacity.
In this example, I set a soft limit of 1GB (1,048,576 KB) and a hard limit of 1.2GB (1,258,291 KB).
Step 5: Adjusting the Grace Period
To change how much time a user has to delete files after hitting their soft limit, use the timer command:
sudo edquota -tI typically set this to 48 hours for production environments. It provides enough time for a developer to react during the work week without leaving the disk dangerously full for too long.
Monitoring Your Storage
Setting limits is only the first step. You also need to see who is nearing their capacity. The repquota command provides a clean summary of every user on the filesystem.
sudo repquota -s /homeThe -s flag converts raw blocks into human-readable MB and GB. I often pipe this output into a weekly cron job to identify the top ten disk consumers across the organization.
Automating the Process
Manually running edquota for every new hire is tedious. If you have a standard 5GB limit for all new developers, use a “prototype” user to clone settings instantly.
sudo edquota -p template_user new_userThis command replicates the limits from template_user to new_user in one second. It ensures consistency and prevents configuration errors during onboarding.
Since implementing these quotas, I haven’t seen a single “No space left on device” error caused by a user. When someone hits their limit, the system stops only their specific write operations. The rest of the OS and the team continue to work without interruption. It is a simple, old-school tool that remains a mandatory part of any professional Linux administrator’s toolkit.
