From 12a57499cbb67e4e08709adf662dedcd6e1e503f Mon Sep 17 00:00:00 2001 From: Lovebug MCP Date: Fri, 22 May 2026 19:04:09 +0000 Subject: [PATCH] =?UTF-8?q?mcp:=20session-notes=20=E2=80=94=20PBS=20NAS=20?= =?UTF-8?q?Off-site=20Backup=20via=20rsync?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- ...05-22-pbs-nas-off-site-backup-via-rsync.md | 81 +++++++++++++++++++ 1 file changed, 81 insertions(+) create mode 100644 Sources/Homelab/2026-05-22-pbs-nas-off-site-backup-via-rsync.md diff --git a/Sources/Homelab/2026-05-22-pbs-nas-off-site-backup-via-rsync.md b/Sources/Homelab/2026-05-22-pbs-nas-off-site-backup-via-rsync.md new file mode 100644 index 0000000..9e12229 --- /dev/null +++ b/Sources/Homelab/2026-05-22-pbs-nas-off-site-backup-via-rsync.md @@ -0,0 +1,81 @@ +--- +created: '2026-05-22' +path: Sources/Homelab +project: herbys-dev-setup +tags: +- proxmox +- pbs +type: session-notes +--- + +## Outcome + +Set up a complete off-site backup pipeline from Proxmox VE through PBS LXC to NAS cold storage, plus validated the full round-trip restore on day one. Both local and DR restore paths tested with a running VM clone. + +Final architecture: +- **PVE** runs daily backup jobs to local PBS datastore `main` at `/pbsstore/datastore` (bind-mounted into LXC 110 as `/datastore`) +- **Weekly systemd timer on PVE host** runs `rsync -a --delete /pbsstore/datastore/ /mnt/nas-pbs/pbs-mirror/` at Sun 04:00, copying chunks to NAS via NFS +- **PBS LXC** never touches the NAS — sidesteps the PBS-in-LXC introspection issue entirely +- **DR procedure validated**: rsync NAS mirror back to `/pbsstore/datastore-test`, register as PBS datastore, restore VM, boot with NIC disconnected to validate + +What's working at session end: +- PBS LXC 110 on Debian 12 (systemd 252), daemons running with correct user separation (api as root, proxy as `backup`) +- Local datastore `main` accepting daily backups at 02:00 +- NAS NFS export mounted on PVE host at `/mnt/nas-pbs`, persistent fstab +- Weekly rsync timer enabled and queued for next Sunday 04:00 +- Maintenance jobs scheduled (prune 03:00, gc 03:30, verify Sat 04:00 on `main`; obsolete `nas-offsite`-targeted jobs still exist and need cleanup since we pivoted away from a NAS-side datastore) +- Test restore datastore `rnas-offsite` at `/pbsstore/datastore-test` kept as validated DR target (frozen at this session's state, will drift from `main`) + +## Topics Covered + +**PBS install gotcha — Debian 13 systemd 257.** Original LXC was built on `debian-13-standard` template. Systemd 257 in this configuration silently failed to honor `User=` directives in PBS unit files. Both `proxmox-backup` and `proxmox-backup-proxy` ran as root, and the proxy refused to start. Symptoms persisted across stop/start cycles and didn't respond to `Type=simple` override. Resolution: destroyed CT 110, rebuilt on `debian-12-standard` (systemd 252). Same VMID, same hostname, same MAC for DHCP consistency. Daemons now run with correct user separation. + +**Bind-mount approach for NFS datastore (rejected).** Mounted NFS on PVE host, bind-mounted into LXC at `/mnt/nas-backup`. `datastore create` succeeded and chunk store populated, but PBS API calls failed with misleading "unable to open chunk store" error. Traced via proxy logs to `find_mounted_device` returning ENOENT — PBS's filesystem introspection can't traverse PVE bind mounts of NFS shares from inside the LXC namespace. + +**Direct NFS in LXC (also rejected).** Pivoted to mounting NFS directly inside the LXC. Required `features: nesting=1,mount=nfs` on the container. Initial mount blocked by `no_root_squash` + UID mismatch — directory was owned by `trucktrav-admin:media` UID 1000/1001, but PBS runs as `backup` UID 34. Resolved by letting PBS chown the path via `no_root_squash`. Mount worked, but PBS still failed with the same `find_mounted_device` ENOENT. Confirms the issue is PBS-in-LXC mount introspection, not the mount method. + +**SMB experiment (also rejected).** Hypothesis: SMB mounts register differently in `/proc/self/mountinfo` and might pass PBS's introspection check. Set up `[pbs-backup]` Samba share on NAS, mounted in LXC with `uid=34,gid=34` (sidesteps NFS's UID-on-the-wire identity problem entirely). Initial mount failed with `vfs_ChDir Permission denied` because PBS had previously chowned the directory to `backup:backup` via `no_root_squash`, blocking the `trucktrav-admin` SMB user. Fixed by chowning back to `trucktrav-admin:media`. Started a `datastore create` over SMB to test — abandoned before completion due to extreme slowness on cold spinners (65k mkdir over SMB). + +**rsync from PVE host (the actual answer).** PBS's chunk store at `/pbsstore/datastore/` is just a directory full of content-addressed files. rsync doesn't care that PBS thinks it owns them — it just copies bytes. PVE host already has NFS mount to NAS at `/mnt/nas-pbs`. Cron/systemd timer runs `rsync -a --delete` between them. PBS never sees the NAS path; the LXC introspection issue is sidestepped entirely. Implementation: systemd `.service` and `.timer` units, weekly Sunday 04:00. Initial bulk sync took hours; subsequent delta-transfers will be minutes. + +**End-to-end restore validation.** Tested both local and DR restore paths with a running VM clone. Created a test datastore at `/pbsstore/datastore-test` (PVE bind-mounted into LXC as `/datastore-test`), rsynced NAS mirror back to it, registered as PBS datastore, restored HerbyDev VM 100 to a throwaway VMID. Booted clone with NIC disconnected (live VM has static IP `10.0.21.207` baked into cloud-init netplan, would conflict on a real network). Verified services, disk usage, no failed units. Restore valid for both local and NAS-rsync-mirror paths. + +## Key Learnings + +- **Debian 12 is the supported PBS base, not Debian 13.** Systemd 257 in D13 silently breaks PBS unit `User=` directives in LXC. Always match the base OS to PBS's officially supported version (bookworm = D12). + +- **PBS-in-LXC + network filesystem datastore is fundamentally broken.** PBS's `find_mounted_device` introspection fails for NFS, SMB, and bind-mounted-NFS inside privileged LXCs — regardless of mount method. Forum consensus is to use a full VM for PBS if NFS datastores are required. We chose the better alternative for our use case: rsync the local datastore externally. + +- **rsync of the chunk store directly is a clean alternative to PBS sync jobs.** PBS chunks are content-addressed immutable files, ideal for rsync delta-transfer. Avoids PBS API complexity, sidesteps LXC introspection issues, works with any destination protocol. Tradeoff: DR restore requires a copy-back step before chunks are usable. Acceptable for off-site insurance scenarios. + +- **`find_mounted_device` errors surface as wildly misleading "no such file" messages.** PBS reports the error against `.chunks/` even though the actual failure is the mount introspection. Always check `journalctl -u proxmox-backup-proxy` for the underlying ENOENT on the mount path itself — that's the real signal. + +- **Don't recreate the NAS-side directory while NFS clients are mounted.** Doing so spawns stale handle cascades that take significant remount work to clear. Always: unmount clients → modify NAS → remount clients. + +- **`exportfs -rav` is mandatory after any `/etc/exports` edit.** Verify with `exportfs -v` — the file and the runtime state can drift, leading to confusing "it worked before" failures. + +- **mergerfs NFS exports require `fsid=`.** FUSE filesystems don't get auto-assigned filesystem IDs, so the export needs an explicit fsid to be stable across restarts. + +- **The `backup` user permission rabbit hole was a red herring.** PBS api daemon runs as root and handles chunk store I/O. Diagnosing whether the `backup` system user could read the chunk store consumed real session time but wasn't the blocker. + +- **Test the restore path on day one.** A backup system you've never restored from is hope, not insurance. The clone-and-boot validation took an hour and converts the entire pipeline from "should work" to "demonstrably works." + +- **Cloud-init netplan injects the live VM's static IP into restored clones.** Booting a clone with NIC connected = ARP conflict with the running original. Disconnect NIC before boot, or edit netplan in the clone before bringing up networking. DR documentation should call this out. + +- **PVE SPICE proxy hostname needs DNS or sed-rewrite.** The `.vv` file PVE generates uses the datacenter hostname (`pve.herbylab.dev`) which doesn't resolve from the tower yet. Workaround: `sed -i 's|pve.herbylab.dev|10.0.21.188|' file.vv` before launching remote-viewer. Real fix is Knot DNS adding an A record for `pve.herbylab.dev`. + +- **Multiple `datastore create` cycles on cold spinners burn hours.** Each create writes 65,536 shard subdirectories. Once you've committed to rebuild, don't do partial state changes — they each cost a full chunkstore init. + +- **NFS is faster than SMB for many-small-files workloads, but the user-mapping pain is real.** For Linux-to-Linux with stable UIDs and trustworthy auth, NFS wins. For mixed environments or where UID drift is a concern, SMB's user-authenticated model with `uid=`/`gid=` mount options sidesteps the entire identity problem. + +## Follow-ons + +- [ ] Remove the obsolete `pj-nas`, `vj-nas` prune/verify jobs and `nas-offsite` gc schedule from PBS — they reference a datastore that no longer exists since the pivot to rsync +- [ ] Decide whether to keep `pbs-nas-test` storage and `rnas-offsite` datastore + `/pbsstore/datastore-test` as a frozen DR validation point, or tear down (currently kept) +- [ ] Add A record for `pve.herbylab.dev → 10.0.21.188` in Knot DNS so SPICE files stop needing sed-rewrite +- [ ] Fix the inconsistent labeling — PVE shows VM 100 as "HerbyDev" but OS hostname is `herbys-dev`. Pick one and align both. +- [ ] Cleanup leftover artifacts on NAS: the abandoned chunk store created during SMB experiment may still have orphan dirs at `/mnt/media-cold/backups/pbs` outside the `pbs-mirror/` subdirectory +- [ ] Test the weekly rsync timer next Sunday — confirm Sun 2026-05-24 04:00 fires, completes, and produces a clean delta transfer (should be minutes, not hours) +- [ ] Consider a `--dry-run` test rsync occasionally to spot drift between source and mirror — would catch any cases where chunks change without timestamps updating +- [ ] Document the DR runbook somewhere visible (vault page or pinned README) so the recovery procedure isn't only in this session note +- [ ] Long-term: consider whether to migrate PBS to a full VM if the use case ever requires native NFS datastore support (e.g., for PBS sync jobs to another PBS instance over the network) \ No newline at end of file