mcp: session-notes — PBS NAS Off-site Backup via rsync
This commit is contained in:
Lovebug MCP
parent
70d71c71ef
commit
12a57499cb
@ -0,0 +1,81 @@
|
|||||||
|
---
|
||||||
|
created: '2026-05-22'
|
||||||
|
path: Sources/Homelab
|
||||||
|
project: herbys-dev-setup
|
||||||
|
tags:
|
||||||
|
- proxmox
|
||||||
|
- pbs
|
||||||
|
type: session-notes
|
||||||
|
---
|
||||||
|
|
||||||
|
## Outcome
|
||||||
|
|
||||||
|
Set up a complete off-site backup pipeline from Proxmox VE through PBS LXC to NAS cold storage, plus validated the full round-trip restore on day one. Both local and DR restore paths tested with a running VM clone.
|
||||||
|
|
||||||
|
Final architecture:
|
||||||
|
- **PVE** runs daily backup jobs to local PBS datastore `main` at `/pbsstore/datastore` (bind-mounted into LXC 110 as `/datastore`)
|
||||||
|
- **Weekly systemd timer on PVE host** runs `rsync -a --delete /pbsstore/datastore/ /mnt/nas-pbs/pbs-mirror/` at Sun 04:00, copying chunks to NAS via NFS
|
||||||
|
- **PBS LXC** never touches the NAS — sidesteps the PBS-in-LXC introspection issue entirely
|
||||||
|
- **DR procedure validated**: rsync NAS mirror back to `/pbsstore/datastore-test`, register as PBS datastore, restore VM, boot with NIC disconnected to validate
|
||||||
|
|
||||||
|
What's working at session end:
|
||||||
|
- PBS LXC 110 on Debian 12 (systemd 252), daemons running with correct user separation (api as root, proxy as `backup`)
|
||||||
|
- Local datastore `main` accepting daily backups at 02:00
|
||||||
|
- NAS NFS export mounted on PVE host at `/mnt/nas-pbs`, persistent fstab
|
||||||
|
- Weekly rsync timer enabled and queued for next Sunday 04:00
|
||||||
|
- Maintenance jobs scheduled (prune 03:00, gc 03:30, verify Sat 04:00 on `main`; obsolete `nas-offsite`-targeted jobs still exist and need cleanup since we pivoted away from a NAS-side datastore)
|
||||||
|
- Test restore datastore `rnas-offsite` at `/pbsstore/datastore-test` kept as validated DR target (frozen at this session's state, will drift from `main`)
|
||||||
|
|
||||||
|
## Topics Covered
|
||||||
|
|
||||||
|
**PBS install gotcha — Debian 13 systemd 257.** Original LXC was built on `debian-13-standard` template. Systemd 257 in this configuration silently failed to honor `User=` directives in PBS unit files. Both `proxmox-backup` and `proxmox-backup-proxy` ran as root, and the proxy refused to start. Symptoms persisted across stop/start cycles and didn't respond to `Type=simple` override. Resolution: destroyed CT 110, rebuilt on `debian-12-standard` (systemd 252). Same VMID, same hostname, same MAC for DHCP consistency. Daemons now run with correct user separation.
|
||||||
|
|
||||||
|
**Bind-mount approach for NFS datastore (rejected).** Mounted NFS on PVE host, bind-mounted into LXC at `/mnt/nas-backup`. `datastore create` succeeded and chunk store populated, but PBS API calls failed with misleading "unable to open chunk store" error. Traced via proxy logs to `find_mounted_device` returning ENOENT — PBS's filesystem introspection can't traverse PVE bind mounts of NFS shares from inside the LXC namespace.
|
||||||
|
|
||||||
|
**Direct NFS in LXC (also rejected).** Pivoted to mounting NFS directly inside the LXC. Required `features: nesting=1,mount=nfs` on the container. Initial mount blocked by `no_root_squash` + UID mismatch — directory was owned by `trucktrav-admin:media` UID 1000/1001, but PBS runs as `backup` UID 34. Resolved by letting PBS chown the path via `no_root_squash`. Mount worked, but PBS still failed with the same `find_mounted_device` ENOENT. Confirms the issue is PBS-in-LXC mount introspection, not the mount method.
|
||||||
|
|
||||||
|
**SMB experiment (also rejected).** Hypothesis: SMB mounts register differently in `/proc/self/mountinfo` and might pass PBS's introspection check. Set up `[pbs-backup]` Samba share on NAS, mounted in LXC with `uid=34,gid=34` (sidesteps NFS's UID-on-the-wire identity problem entirely). Initial mount failed with `vfs_ChDir Permission denied` because PBS had previously chowned the directory to `backup:backup` via `no_root_squash`, blocking the `trucktrav-admin` SMB user. Fixed by chowning back to `trucktrav-admin:media`. Started a `datastore create` over SMB to test — abandoned before completion due to extreme slowness on cold spinners (65k mkdir over SMB).
|
||||||
|
|
||||||
|
**rsync from PVE host (the actual answer).** PBS's chunk store at `/pbsstore/datastore/` is just a directory full of content-addressed files. rsync doesn't care that PBS thinks it owns them — it just copies bytes. PVE host already has NFS mount to NAS at `/mnt/nas-pbs`. Cron/systemd timer runs `rsync -a --delete` between them. PBS never sees the NAS path; the LXC introspection issue is sidestepped entirely. Implementation: systemd `.service` and `.timer` units, weekly Sunday 04:00. Initial bulk sync took hours; subsequent delta-transfers will be minutes.
|
||||||
|
|
||||||
|
**End-to-end restore validation.** Tested both local and DR restore paths with a running VM clone. Created a test datastore at `/pbsstore/datastore-test` (PVE bind-mounted into LXC as `/datastore-test`), rsynced NAS mirror back to it, registered as PBS datastore, restored HerbyDev VM 100 to a throwaway VMID. Booted clone with NIC disconnected (live VM has static IP `10.0.21.207` baked into cloud-init netplan, would conflict on a real network). Verified services, disk usage, no failed units. Restore valid for both local and NAS-rsync-mirror paths.
|
||||||
|
|
||||||
|
## Key Learnings
|
||||||
|
|
||||||
|
- **Debian 12 is the supported PBS base, not Debian 13.** Systemd 257 in D13 silently breaks PBS unit `User=` directives in LXC. Always match the base OS to PBS's officially supported version (bookworm = D12).
|
||||||
|
|
||||||
|
- **PBS-in-LXC + network filesystem datastore is fundamentally broken.** PBS's `find_mounted_device` introspection fails for NFS, SMB, and bind-mounted-NFS inside privileged LXCs — regardless of mount method. Forum consensus is to use a full VM for PBS if NFS datastores are required. We chose the better alternative for our use case: rsync the local datastore externally.
|
||||||
|
|
||||||
|
- **rsync of the chunk store directly is a clean alternative to PBS sync jobs.** PBS chunks are content-addressed immutable files, ideal for rsync delta-transfer. Avoids PBS API complexity, sidesteps LXC introspection issues, works with any destination protocol. Tradeoff: DR restore requires a copy-back step before chunks are usable. Acceptable for off-site insurance scenarios.
|
||||||
|
|
||||||
|
- **`find_mounted_device` errors surface as wildly misleading "no such file" messages.** PBS reports the error against `.chunks/` even though the actual failure is the mount introspection. Always check `journalctl -u proxmox-backup-proxy` for the underlying ENOENT on the mount path itself — that's the real signal.
|
||||||
|
|
||||||
|
- **Don't recreate the NAS-side directory while NFS clients are mounted.** Doing so spawns stale handle cascades that take significant remount work to clear. Always: unmount clients → modify NAS → remount clients.
|
||||||
|
|
||||||
|
- **`exportfs -rav` is mandatory after any `/etc/exports` edit.** Verify with `exportfs -v` — the file and the runtime state can drift, leading to confusing "it worked before" failures.
|
||||||
|
|
||||||
|
- **mergerfs NFS exports require `fsid=`.** FUSE filesystems don't get auto-assigned filesystem IDs, so the export needs an explicit fsid to be stable across restarts.
|
||||||
|
|
||||||
|
- **The `backup` user permission rabbit hole was a red herring.** PBS api daemon runs as root and handles chunk store I/O. Diagnosing whether the `backup` system user could read the chunk store consumed real session time but wasn't the blocker.
|
||||||
|
|
||||||
|
- **Test the restore path on day one.** A backup system you've never restored from is hope, not insurance. The clone-and-boot validation took an hour and converts the entire pipeline from "should work" to "demonstrably works."
|
||||||
|
|
||||||
|
- **Cloud-init netplan injects the live VM's static IP into restored clones.** Booting a clone with NIC connected = ARP conflict with the running original. Disconnect NIC before boot, or edit netplan in the clone before bringing up networking. DR documentation should call this out.
|
||||||
|
|
||||||
|
- **PVE SPICE proxy hostname needs DNS or sed-rewrite.** The `.vv` file PVE generates uses the datacenter hostname (`pve.herbylab.dev`) which doesn't resolve from the tower yet. Workaround: `sed -i 's|pve.herbylab.dev|10.0.21.188|' file.vv` before launching remote-viewer. Real fix is Knot DNS adding an A record for `pve.herbylab.dev`.
|
||||||
|
|
||||||
|
- **Multiple `datastore create` cycles on cold spinners burn hours.** Each create writes 65,536 shard subdirectories. Once you've committed to rebuild, don't do partial state changes — they each cost a full chunkstore init.
|
||||||
|
|
||||||
|
- **NFS is faster than SMB for many-small-files workloads, but the user-mapping pain is real.** For Linux-to-Linux with stable UIDs and trustworthy auth, NFS wins. For mixed environments or where UID drift is a concern, SMB's user-authenticated model with `uid=`/`gid=` mount options sidesteps the entire identity problem.
|
||||||
|
|
||||||
|
## Follow-ons
|
||||||
|
|
||||||
|
- [ ] Remove the obsolete `pj-nas`, `vj-nas` prune/verify jobs and `nas-offsite` gc schedule from PBS — they reference a datastore that no longer exists since the pivot to rsync
|
||||||
|
- [ ] Decide whether to keep `pbs-nas-test` storage and `rnas-offsite` datastore + `/pbsstore/datastore-test` as a frozen DR validation point, or tear down (currently kept)
|
||||||
|
- [ ] Add A record for `pve.herbylab.dev → 10.0.21.188` in Knot DNS so SPICE files stop needing sed-rewrite
|
||||||
|
- [ ] Fix the inconsistent labeling — PVE shows VM 100 as "HerbyDev" but OS hostname is `herbys-dev`. Pick one and align both.
|
||||||
|
- [ ] Cleanup leftover artifacts on NAS: the abandoned chunk store created during SMB experiment may still have orphan dirs at `/mnt/media-cold/backups/pbs` outside the `pbs-mirror/` subdirectory
|
||||||
|
- [ ] Test the weekly rsync timer next Sunday — confirm Sun 2026-05-24 04:00 fires, completes, and produces a clean delta transfer (should be minutes, not hours)
|
||||||
|
- [ ] Consider a `--dry-run` test rsync occasionally to spot drift between source and mirror — would catch any cases where chunks change without timestamps updating
|
||||||
|
- [ ] Document the DR runbook somewhere visible (vault page or pinned README) so the recovery procedure isn't only in this session note
|
||||||
|
- [ ] Long-term: consider whether to migrate PBS to a full VM if the use case ever requires native NFS datastore support (e.g., for PBS sync jobs to another PBS instance over the network)
|
||||||
Loading…
Reference in New Issue
Block a user