How a stale git fsmonitor daemon made my agent commits silently empty

A merge request failed CI, I fixed two files with a coding agent, ran git add + git commit + git push, and the push contained nothing. Git reported “nothing to commit, working tree clean” while the fixed files sat in the working tree with different content than HEAD. The commit had silently recorded zero files.

The Problem

The setup: a large monorepo, a linked git worktree, core.fsmonitor=true set globally in ~/.gitconfig, and pi doing the edits. The sequence that triggered it:

1. The MR branch was rebased on the GitLab server side.
2. Locally I synced with git reset --hard origin/<branch> (fast, correct, index consistent).
3. The agent edited two Java test files through its edit tool.
4. git add <files> && git commit ran the pre-commit hooks, which saw 0 staged files, and git answered “nothing to commit, working tree clean”.

The contradiction was provable from three commands:

# Working tree really has the new content:
$ git hash-object backend/.../ContractPricingPropertyTest.java
19aa341574a3e6897b7a8096da3824435d600cce

# Index and HEAD still hold the old blob:
$ git ls-files -s backend/.../ContractPricingPropertyTest.java
100644 4dc5c7db6e3114871ab65b83a8f3dd6e88d541b9 0    ...

# Yet git sees no difference at all:
$ git diff HEAD --stat
# (empty)

git hash-object reads the file directly and bypasses the index, so the working tree was telling the truth. git add, which runs through the index, stored the old hash anyway. That means git was deciding the file was unchanged without ever hashing it.

Two usual suspects were ruled out first:

• git ls-files -v showed H for both files: no assume-unchanged or skip-worktree bits.
• git check-attr and git config --get-regexp '^filter\.': no clean/smudge filters rewriting content.

The index retained CE_FSMONITOR_VALID bits for the edited paths because the fsmonitor daemon returned an empty change list. Git trusted those bits, skipped lstat(2), and add, status, and diff all agreed with a lie.

Root Cause: inotify Watch Exhaustion Across Worktrees

core.fsmonitor trades hashing work for a kernel event stream. A background daemon (git-fsmonitor—daemon) watches the worktree. Git hands the daemon a token from its index and asks what changed since that token, then only re-stats and re-hashes paths the daemon reports. Git marks clean entries in the index with the CE_FSMONITOR_VALID bit and skips lstat(2) calls for everything else. GitHub’s engineering writeup on FSMonitor notes the central requirement: the daemon must continuously maintain a complete view of all filesystem changes between git commands.

On Linux, this mechanism relies on the kernel inotify subsystem. Unlike macOS (where FSEvents provides recursive directory watches from the repository root), Linux inotify is non-recursive. The daemon must attach an individual watch to every single directory in the repository tree.

This architecture fails in multi-worktree agent environments for three reasons:

1. Watch multiplication across worktrees. Every linked git worktree spawns its own isolated git-fsmonitor--daemon process. A monorepo with 130,000 directories requires 130,000 inotify watches per worktree. Three active worktrees demand 390,000 watches.
2. Silent watch exhaustion (ENOSPC). When the total watch count reaches /proc/sys/fs/inotify/max_user_watches (set to 308,945 on this machine), inotify_add_watch calls fail silently. The daemon continues running, but deeper directories never receive watches. When the agent edited files in backend/src/test/java/..., the kernel emitted no events to the daemon. The daemon answered “nothing changed”, and Git trusted it.
3. Sub-second agent execution bursts. Agents execute commands without human pauses. Running git reset --hard, overwriting files, and invoking git add within milliseconds widens the race window between index tokens and inotify event queues.

When a human programmer runs git status and misses a file, they pause and investigate. An automated coding agent trusts the empty status, reports completion, and pushes an empty commit.

What Changed

To unblock the immediate commit, I stopped the daemon:

git fsmonitor--daemon stop   # next git command rescans disk

Stopping the daemon forced the subsequent git status to perform a full working-tree scan. Both modified files appeared immediately, and git add staged the updated blobs.

To resolve this permanently in ~/.dotfiles, I replaced the global fsmonitor setting with the recommended stat-based configuration:

# ~/.dotfiles/.gitconfig
[core]
    fsmonitor = false
    untrackedCache = true
    preloadIndex = true

I terminated the lingering orphan daemon fleet with pkill -f "git fsmonitor--daemon", immediately freeing ~300,000 inotify watches back to the kernel.

Why Stat-Based Caching Works

To understand why this fix works (and why Git provides these knobs), it helps to examine how Git checks for file changes under the hood.

The Stat-Dirty Mechanism

When checking whether a working-tree file has changed, reading and hashing every file’s contents with SHA-1 or SHA-256 would make git status unbearably slow.

Instead, Git’s index (.git/index) caches filesystem metadata alongside each blob hash:

• mtime (modification time, down to the nanosecond)
• ctime (metadata change time)
• file_size (size on disk)
• ino and dev (inode and device number)
• mode (file permissions)

When Git runs status, diff, or add, it issues an lstat(2) syscall on the file path. Through ce_match_stat(), it compares the returned struct stat against the index entry. If mtime, file_size, ino, and mode match, the file is stat-clean: Git knows the content was untouched and skips reading or hashing the file entirely. Only when an attribute differs does Git mark the entry stat-dirty, open the file, and compute its hash.

fsmonitor short-circuits this check by setting the in-memory CE_FSMONITOR_VALID bit on index entries, telling Git to skip calling lstat(2) altogether. When the daemon missed events due to inotify exhaustion, Git trusted that bit and accepted an out-of-date index.

Parallel Preload: Linus Torvalds and preload-index.c

The alternative to trusting a background daemon is parallelizing the stat checks. In 2008, Linus Torvalds introduced preload-index.c in Git with the commit message:

“This can do the lstat() storm in parallel, giving potentially much improved performance for cold-cache cases or things like NFS that have weak metadata caching.”

preload_index() works by dividing the index array into equal slices and launching worker POSIX threads (pthreads):

1. Worker threads run in parallel. Each thread loops over its assigned chunk of index entries, issuing concurrent lstat(2) syscalls.
2. Flagging clean files. When an entry matches its cached stat metadata on disk, the thread marks it with an internal bitflag: CE_UPTODATE.
3. The main thread skip. When worker threads join, Git’s main loop walks the index sequentially as usual. But when it encounters an entry with CE_UPTODATE, it skips calling lstat(2) entirely because the worker thread already verified it.
4. VFS cache warming. For files that are dirty or untracked, the parallel lstat(2) storm pre-populates the OS dentry and inode caches across all CPU cores before the main thread requests them.

In Git 2.1.0 (2014), core.preloadIndex was made true by default because multi-core CPUs made thread-spawning overhead negligible, while eliminating major filesystem latency bottlenecks (especially on Windows, where GetFileAttributesExW has higher syscall overhead than POSIX lstat).

Directory Caching: core.untrackedCache

While preloadIndex accelerates tracked files, Git also needs to detect untracked files without crawling every folder.

core.untrackedCache stores directory mtime values inside an extension block of .git/index. Because creating, deleting, or renaming a file in a directory updates that directory’s own mtime, Git can compare the directory timestamp with its cached value. If the folder mtime matches, Git skips opendir() and readdir() on that entire subtree, bypassing .gitignore matching for thousands of files.

Architectural Tradeoffs: Stateful Daemons vs. Stateless Polling

The incident highlights a core architectural tension in version control tooling:

For a single human workstation on macOS with a monolithic repo containing over a million files, core.fsmonitor provides genuine value by dropping status times from ~1.5 seconds down to ~20 milliseconds.

For multi-worktree Linux environments and high-concurrency coding agents, the calculus is inverted. Modern NVMe drives on PCIe 4.0/5.0 can parallel-stat 100,000 files in under 40 milliseconds. Sacrificing deterministic stage accuracy for a 20ms speedup introduces an unacceptable failure mode when automated agents trust empty statuses.

Best Practices for Agent Monorepos

• Disable global fsmonitor. Keep core.fsmonitor=false in ~/.gitconfig. The ~20–40ms speedup is not worth silent empty commits.
• Combine untrackedCache with preloadIndex. core.untrackedCache skips clean directory traversals, while core.preloadIndex parallelizes lstat(2) across CPU cores.
• Use sparse checkouts for massive trees. If a monorepo exceeds 100,000 directories, configure git sparse-checkout set <paths> in cone mode. Scoping the agent to the backend cuts directory count from 130,000 down to 2,000, eliminating inotify pressure entirely.
• Scope fsmonitor locally when required. If a specific multi-million-file repository requires fsmonitor, enable it only locally with git config --local core.fsmonitor true and raise fs.inotify.max_user_watches above 1,000,000 in /etc/sysctl.conf.

Results

• I replaced the silent no-op commit with the real fix (df0e7f48f), and the MR pipeline turned green.
• All orphaned background daemons were terminated, releasing ~300,000 kernel inotify watches.
• Global git configuration now relies on core.untrackedCache and core.preloadIndex, guaranteeing 100% stage accuracy across concurrent agent worktrees.

References

• pi: minimalistic agent harness used for edits
• preload-index.c in Git: parallel index preload implementation and CE_UPTODATE mechanics
• Add cache preload facility (commit 8431e21): Linus Torvalds’ 2008 commit introducing parallel index preloading
• git-fsmonitor—daemon(1): official docs on the daemon, tokens, and startup scan
• inotify(7): Linux file system event monitoring and max_user_watches limits
• lstat(2): POSIX file status system call
• Improve Git monorepo performance with a file system monitor: GitHub Engineering on enabling FSMonitor and its completeness requirement
• pre-commit #2795: index misreporting state until fsmonitor is disabled
• lazygit #5906: index.lock contention from concurrent tooling, a neighboring failure mode