Posted inLinux

Real-Time File System Monitoring with inotifywait and incron on Linux: Auto-Trigger Scripts on File Events

Linux tutorial - IT technology blog
Linux tutorial - IT technology blog

The Problem: You Can’t Watch Every File Manually

Picture this: a deployment script drops a config file into /etc/app/, but the service doesn’t reload. Or a user uploads a file to /var/uploads/ and nothing happens until someone manually kicks off a processing job. You end up either polling with a cron job every minute — wasteful and laggy — or writing complex daemon code from scratch.

After managing 10+ Linux VPS instances over three years, I found that reactive automation — acting the moment a file changes — is far more reliable than scheduled polling. The Linux kernel already tracks every filesystem event. You just need the right tools to tap into it.

Enter inotify, inotifywait, and incron. React to file events instantly, no daemon code required.

Core Concepts: How Linux Tracks File Events

The inotify Kernel Subsystem

Introduced in Linux kernel 2.6.13, inotify (inode notify) lets processes watch files and directories for specific events — no polling, no timers. The kernel pushes notifications the instant something changes.

Supported events include:

  • IN_CREATE — a file or directory was created
  • IN_MODIFY — a file was written to
  • IN_DELETE — a file or directory was deleted
  • IN_MOVED_FROM / IN_MOVED_TO — file was renamed or moved
  • IN_CLOSE_WRITE — a writable file was closed (very useful for detecting completed uploads)
  • IN_ATTRIB — metadata changed (permissions, ownership, timestamps)

inotifywait: Command-Line Access to inotify

inotifywait ships as part of the inotify-tools package. It blocks until an event occurs, prints the event, then exits — or keeps looping with the -m flag. Wrap it in a while loop and you have a simple, scriptable file watcher.

incron: inotify + cron = incron

Think of incron as cron, but triggered by filesystem events rather than time. Define rules in an incrontab file, and incrond runs your command automatically when the event fires. No shell loop, no manual restarts — it runs as a daemon in the background.

Hands-On Practice

Step 1: Install the Tools

On Debian/Ubuntu:

sudo apt update
sudo apt install inotify-tools incron

On RHEL/AlmaLinux/Rocky:

sudo dnf install inotify-tools incron

Step 2: Quick Test with inotifywait

Before wiring up any automation, always verify events manually. Open two terminals.

Terminal 1 — start watching:

inotifywait -m -r -e create,modify,delete /tmp/watch-test/

Terminal 2 — trigger some events:

mkdir -p /tmp/watch-test
touch /tmp/watch-test/hello.txt
echo "data" >> /tmp/watch-test/hello.txt
rm /tmp/watch-test/hello.txt

You’ll see output like:

/tmp/watch-test/ CREATE hello.txt
/tmp/watch-test/ MODIFY hello.txt
/tmp/watch-test/ DELETE hello.txt

This confirms the kernel is reporting events correctly for that directory. I’ve been burned before by a symlink pointing somewhere inotify wasn’t actually watching — this quick test catches that before anything breaks in production.

Step 3: Write a Shell Loop Watcher with inotifywait

For lightweight, one-off automation, a shell script is the fastest path forward:

#!/bin/bash
# watch-uploads.sh — process files as soon as they arrive

WATCH_DIR="/var/uploads"
PROCESS_SCRIPT="/opt/scripts/process-file.sh"

inotifywait -m -e close_write --format '%f' "$WATCH_DIR" | while read FILENAME; do
    echo "[$(date)] Detected: $FILENAME"
    "$PROCESS_SCRIPT" "$WATCH_DIR/$FILENAME"
done

Key tip: use close_write instead of modify for uploaded files. The modify event fires on every write syscall — copy a 500MB file and you’ll get hundreds of events mid-transfer. close_write fires exactly once, when the file handle closes after writing. Much cleaner.

Run this script as a systemd service so it survives reboots:

# /etc/systemd/system/watch-uploads.service
[Unit]
Description=Watch /var/uploads for new files
After=network.target

[Service]
ExecStart=/opt/scripts/watch-uploads.sh
Restart=on-failure
User=www-data

[Install]
WantedBy=multi-user.target
sudo systemctl daemon-reload
sudo systemctl enable --now watch-uploads

Step 4: Set Up incron for System-Level File Triggers

incron shines for persistent, daemon-managed rules — config reloads, log rotation triggers, or multi-user setups where each user manages their own incrontab.

First, allow your user (or root) to use incron:

echo "root" | sudo tee -a /etc/incron.allow

Edit the incrontab for root:

sudo incrontab -e

Syntax: <path> <event(s)> <command>

# Reload nginx when config changes
/etc/nginx/conf.d IN_CLOSE_WRITE systemctl reload nginx

# Compress any new log file dropped into /var/log/archive/
/var/log/archive IN_CREATE gzip $@/$#

# Alert when a file is deleted from a sensitive directory
/etc/cron.d IN_DELETE /opt/scripts/alert-cron-delete.sh $#

Special incron variables:

  • $@ — the watched directory path
  • $# — the filename that triggered the event
  • $$ — a literal $ sign
  • $% — the event flags as a string

Enable and start incrond:

sudo systemctl enable --now incrond
sudo systemctl status incrond

Step 5: Watching Recursively and Filtering Events

One common gotcha: incron does not watch subdirectories by default. For recursive watching, inotifywait -r is the better choice. With incron, you either add a rule per subdirectory or write a wrapper that dynamically registers new watches.

Filtering by filename pattern in inotifywait is easy with grep:

inotifywait -m -e close_write --format '%f' /var/uploads | \
  grep --line-buffered '\.csv$' | \
  while read FILENAME; do
    /opt/scripts/import-csv.sh "/var/uploads/$FILENAME"
done

Don’t forget --line-buffered on grep. Without it, grep buffers output in chunks and your script appears to hang waiting. This trips up almost everyone the first time they pipe inotifywait output through grep.

Step 6: Debouncing Rapid Events

Some editors and tools write files in multiple steps — write a temp file, rename it, delete the original. A single logical save generates a burst of events. Here’s a simple debounce in bash:

#!/bin/bash
WATCH_DIR="/etc/app"
DEBOUNCE=2  # seconds to wait after last event

inotifywait -m -e close_write,moved_to --format '%f' "$WATCH_DIR" | \
while read FILENAME; do
    # Cancel pending reload if one is queued
    [ -n "$DEBOUNCE_PID" ] && kill "$DEBOUNCE_PID" 2>/dev/null
    (
        sleep $DEBOUNCE
        echo "[$(date)] Reloading due to change: $FILENAME"
        systemctl reload myapp
    ) &
    DEBOUNCE_PID=$!
done

This waits 2 seconds after the last event before triggering the action. During a rolling deployment that rewrites a config file 4–5 times in quick succession, you get exactly one reload instead of five.

Practical Tip: Check Your inotify Watch Limits

Each inotify watch consumes a kernel resource. The default limit is 8,192 watches per user. On a server with many watched directories, you can hit that ceiling fast:

# Check current limit
cat /proc/sys/fs/inotify/max_user_watches

# Increase it temporarily
sudo sysctl fs.inotify.max_user_watches=524288

# Make it permanent
echo "fs.inotify.max_user_watches=524288" | sudo tee -a /etc/sysctl.d/99-inotify.conf
sudo sysctl -p /etc/sysctl.d/99-inotify.conf

This limit is easy to underestimate. On one VPS running both a Node.js app (which uses inotify heavily for hot-reloading) and our incron rules, we silently hit the ceiling. Incron stopped triggering with zero error output — just silence. Running dmesg | grep inotify revealed the culprit immediately. Always check this before deploying to production.

When to Use Which Tool

  • inotifywait in a loop: Quick scripts, development automation, single-purpose watchers. Easy to test interactively.
  • incron: System-level rules you want managed as a daemon. Ideal for ops tasks like auto-reloading services or compressing logs. All rules live in one auditable incrontab.
  • Python’s watchdog library: Cross-platform file watching inside a Python application. It wraps inotify with a clean API and handles platform differences transparently.

Putting inotify to Work

Switching from cron-every-minute polling to event-driven file watching cuts latency from up to 60 seconds down to milliseconds. It also eliminates unnecessary CPU wake-ups — the process sleeps until the kernel signals it.

Start small. Pick one directory, watch it with inotifywait -m for a day, and observe the events. You’ll spot automation opportunities fast — config files that need reloading, uploads waiting on a processing job, sensitive paths that should never change silently.

Between inotifywait for scripting and incron for daemon-managed rules, you can cover 90% of real-world file monitoring needs without writing a single line of C or deploying a heavyweight monitoring agent.

Share:
Tags: