Self-Healing Backupautonomous infrastructure
AI & Automation · autonomous infrastructure

The backup that re-architected itself

Anyone can wire up a happy path. The judgment shows when it breaks in production— and this one broke in a genuinely hard-to-diagnose way.

I built a backup that runs unattended on four triggers and captures my Claude setup, my dotfiles, and my whole Mac — then debugged it through a filesystem deadlock and re-architected it mid-flight. It’s the clearest proof that I don’t just use AI tooling: I build the infrastructure around it and fix it when it fails.

4 triggers → one idempotent push
loginwakesleepnoonatomic lockGitHub · LFSretries next run
Autonomous infrastructureBash · launchd · Git LFSIdempotent + self-healingRoot-cause debuggingRuns in production, daily
01What it is

An operating layer that runs while I’m asleep.

Not a cron job that copies a folder. A single-authority backup engine that stages my Claude memory and config, dotfiles, launch agents, and full Mac state — keychain, iMessage, SSH, Documents — and pushes it to GitHub over Git LFS, once a day, no matter how it’s triggered.

Four triggers, one target

Login, wake (via sleepwatcher), and a noon fallback all converge on the same idempotent run.

Idempotent per day

A dated marker + a “commit only if changed” guard make any redundant trigger a silent no-op.

Self-healing

Declares success only after verifying the push landed; otherwise it fails loudly and retries next run.

Observable

A durable status stamp and staged failure notifications name the exact step that broke.

!The signature story

“Resource deadlock avoided.” The error pointed at git. Git wasn’t the cause.

The obvious design worked — until my Desktop and Documents became live Google Drive mirror roots. Now the backup was staging into the Drive tree, with the git repo living under it. DriveFS and git fought over the same files and wedged the filesystem: Resource deadlock avoided · mmap failed.

The whole thing looked like a git problem. Chasing that would’ve been a dead end. The real trigger was somewhere else entirely.
02The diagnosis

Two bugs, one disguise.

Under the Drive symptom sat a second, unrelated bug: launchd starts jobs with a bare PATH that excludes Homebrew, so git-lfs was invisible, the LFS hook failed, and git checkoutreturned non-zero. The fix is one line — but finding it meant refusing the first plausible cause.

the PATH fix

# launchd starts jobs with a bare PATH that
# excludes Homebrew — so git-lfs was invisible
# and git checkout returned non-zero.
export PATH="/opt/homebrew/bin:$PATH"

The lesson isn’t the line. It’s that the symptom said “Drive,” the error said “git,” and the cause was the environment.

03The re-architecture

I removed Drive from the path entirely.

Instead of layering workarounds on a doomed design, I subtracted the cause: drop Google Drive from the backup path and write straight to GitHub from each real source. Then I hardened it to survive running constantly — because it fires far more often, and messier, than a happy path expects.

the concurrency guard

# fires on every trigger; two overlapping runs
# corrupt the repo. mkdir is atomic — one wins.
if mkdir "$LOCK" 2>/dev/null; then
    trap 'rmdir "$LOCK"' EXIT
else exit 0  # someone else holds it
fi
Atomic lock

mkdiris atomic, so exactly one run wins and the rest exit clean — no corrupted index.

Marker moved out

The once-a-day marker lived under ~/.claude, which the CLI actively manages — and kept scrubbing it. Moved it to the unmanaged home root.

04The one that hid for six weeks

The worst failure is the one that stays quiet.

Later, the GitHub-push half silently failed for roughly six weeks. launchd lacked Full Disk Access to read ~/Desktop, so it errored with Operation not permitted— but the old Drive snapshot still worked and the error was swallowed by a stray 2>/dev/null. I only caught it restoring onto a new machine.

Root-caused it to a permissions gap, granted Full Disk Access, and hardened it with logging and push-verified markers — so a silent failure can’t hide again.
05What it taught me

Three rules I now design by.

Subtraction beats patching

Removing Drive from the path fixed more than any workaround stacked on top of it ever could.

Symptoms lie

The failure wore git’s face but lived in the environment. Refuse the first plausible cause.

Design for repetition

Idempotency, locks, and verification are what let a nondeterministic system do unattended work you can trust.

This is DevOps-grade reasoning — idempotency, atomic locking, staged failure semantics, observability, a root-caused OS-level deadlock — authored by a product designer.
The point

I build with AI as infrastructure— and I’m the one on call when it breaks.

The same judgment I bring to product design, applied to the tools I build to do that work: knowing when to trust the system, when to guard it, and how to find the real cause when the obvious one is wrong.

← How I approach AI work