The Mac had been offline for a while. When it reconnected to Syncthing at 21:03 UTC on a Saturday night, the LXC’s puller took less than fifteen minutes to start applying the delta: 2,657 file deletions, 574 frontmatter prunes stripping 74,118 lines against just 809 additions, and 166 Apple-style <name> 2.md conflict copies scattered across half the vault.
Nothing was wrong with the receiving machine — not the filesystem, not the sync daemon, not my code. Syncthing did exactly what a sync daemon should: it propagated the state the Mac claimed was true. The trouble was that a few hours earlier, a vault-cleanup tool had manufactured that state while running over a half-evicted, iCloud-backed directory. To the cleanup tool, two thousand stub notes and seventy-four thousand wikilinks simply didn’t exist.
The old setup
Three peers held the vault, and iCloud sat in the middle where it didn’t belong.
The Mac ran the canonical Obsidian client, with the vault living inside Obsidian Mobile’s iCloud container. The iPhone ran against that same container — open the app and the vault was just there, at the cost of putting iCloud Drive in the data path. A Proxmox LXC ran the headless side (the kg tools, the morning briefing, the dispatch agent) and synced with the Mac over Syncthing. Syncthing moved bytes between Mac and LXC; it never touched the phone.
iCloud’s job is to manage storage, not to sync faithfully. On a phone under storage pressure, iOS quietly evicts file contents and leaves a placeholder that reports “not locally present” until iCloud re-downloads it.
flowchart TB subgraph Before["Before: iCloud in the data path"] P1[iPhone] <--> IC[(iCloud — evicts<br/>file contents)] IC <--> M1[Mac] M1 --> CT[cleanup tool reads<br/>evicted files as deleted] CT -->|Syncthing ships<br/>2,657 deletions| L1[LXC] end subgraph After["After: LXC-canonical star + alarms"] M2[Mac] <-->|Syncthing| L2[LXC hub — canonical] P2[iPhone — 3rd-party client] <-->|Syncthing| L2 H2[HAOS] <-->|Syncthing| L2 D2[Docker host] <-->|Syncthing| L2 L2 --> AL[delete-rate alarm<br/>100 in 5 min → Discord] L2 --> ZS[ZFS snapshots] end
What actually happened
By that evening, the Mac’s local view of the vault was incomplete. About 2,600 Stubs/*.md files — each a sub-100-byte index pointer the kg tools use as link targets — weren’t on disk, because iCloud had evicted them and the Mac, with Optimize Storage on, had honored the eviction. To anything walking the directory, the files were gone.
Then a cleanup tool ran. The signature is unmistakable in the diff: 574 notes had their uses:, references:, and similar frontmatter fields pruned, with deletions landing exactly on wikilink targets that no longer resolved. Two thousand six hundred stubs were deemed orphaned and removed.
I never pinned down which tool, or when, or on which device — the Obsidian Linter and my kg:fix-conformance passes will both happily prune references to files they can’t see. But the trigger turned out to be the replaceable half of the problem. A cleanup tool can only do this much damage when the storage layer has already lied about what’s on disk. Take away the lie and the exact tool stops mattering.
Then the Mac’s Syncthing daemon came back online. It saw a folder that had drifted from the LXC’s, computed the delta, and shipped it. The puller peaked at 252 file mutations in a single minute. By morning the working tree had 3,452 dirty paths, none of them authored on that machine — I noticed at 08:30, when my git status scrolled for two screens.
What saved me
One piece of structural luck: I commit the vault. Not on a schedule — just whenever I notice — but the last commit before the sync was a few hours old, and nothing had committed during the destructive delta. The dirty tree was all unstaged, so git checkout HEAD -- on the affected paths rolled the LXC back cleanly.
The Mac’s vault was the opposite. The corruption originated there, the iPhone’s iCloud view was upstream of it, and iCloud keeps no per-edit history I can reach. So I made a blunt call: declare the LXC canonical and delete the Mac and iPhone copies entirely. The LXC was the only peer I’d verified and the only one whose history I could audit, so Mac and phone would rebuild from it.
That bought a clean state to rebuild from. It didn’t touch the architecture that let the corruption happen — iCloud was still in the data path on both clients. Fixing that was the next day’s problem.
The fix I reached for, and dropped
Standing in the wreckage, the fix looked obvious: get iCloud out of the data path with something I controlled end to end. The same day I picked Obsidian LiveSync, a plugin that syncs through a self-hosted CouchDB backend, and started building.
The appeal was that none of it was Apple’s to evict. CouchDB would run on its own LXC, reachable only over Tailscale, with the vault at a normal local path on every device. The database was disposable — a sync index, not the source of truth. What I actually cared about was the phone: LiveSync splits files into content-defined chunks and replicates them as small documents, which was the theorized way to fit a vault this size under the iPhone’s ceiling. Mac↔CouchDB↔LXC even came up and replicated on the first try.
Then I pointed the iPhone at it, and it came apart in three stages. Setup wouldn’t finish — the plugin threw an InvalidStateError from an IndexedDB lifecycle race, on a vault of roughly 5,000 documents and 3.9 GB. When init did stagger through, the storage phase reported DONE:308, FAILED:3327 — a 91% write-failure rate against Obsidian’s iOS sandbox. Then Obsidian Mobile dropped into a crash-freeze-restart loop.
I turned every throttling knob the plugin exposed and none of it moved anything, because the failure was upstream of sync entirely. It was at plugin init. Deltas can’t help you when the database never opens.
That killed the whole architecture, not just the phone leg. The phone was LiveSync’s entire reason to exist in my setup; without it, CouchDB-on-an-LXC is just Syncthing with more moving parts and a database to babysit. So the next day I tore it out. The CouchDB LXC is gone, the plugin is uninstalled, and Mac and LXC are back on Syncthing — but pointed at a plain local path, not the iCloud container. That was always the real fix. iCloud is out of the data path by being removed, not by being swapped for something cleverer.
The topology now
The iPhone rejoined the same day as its own spoke, running a third-party Syncthing-compatible client and paired only with the LXC. The shape is a star, not a mesh: the LXC is the hub, and the Mac, the HAOS box, the Docker host, and the phone are all spokes. Nothing pairs peer-to-peer. If a spoke ever ships a bad delta again, it funnels through the hub instead of fanning out to every peer at once.
And the hub is watched now. The git commit that saved me was luck, so I replaced it with insurance I don’t have to remember to take out. Syncthing already ships its file operations to VictoriaLogs, so the detector is an alert rule over that stream: it counts action:delete events in a rolling five-minute window and pages me through Grafana-to-Discord when the count crosses 100. ZFS snapshots on the LXC’s rootfs give whole-filesystem rollback with no commit-log noise, and Forgejo still sits underneath it all with the canonical-2026-05-17 anchor tag marking the state I rolled back to.
Apple-style 2.md files, by the way, are back as a signal I trust. Syncthing writes its own .sync-conflict-... copies when peers disagree, and that’s routine. But a 2.md still means iCloud got its hands on something it shouldn’t have. That fingerprint didn’t change.
What I’d do differently
The advice my first draft was walking toward was backwards. It was going to tell you to trade a boring sync daemon for a richer, self-hosted one. Instead the richer tool bit me inside a day, at plugin init, on the one device the whole design existed for — before I’d learned anything about how it behaves in steady state.
The lesson that held is duller and truer. Pick the boring, well-understood tool while the replacement is unproven, and build the detection and rollback floors around it yourself. Don’t retire a failure mode you understand for one you haven’t watched survive contact with the hardest case — which, here, was always the phone. I should have tested the riskiest leg before tearing down the one that already worked.
The next time a sync daemon faithfully ships a disaster, I want the recovery to take an afternoon, not a postmortem. This time, I think it will.