petal-power/wiki-vault
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
5d4db2fcdd
wiki-vault/Sources/Homelab/2026-05-22-pbs-nas-off-site-backup-via-rsync.md

9.7 KiB
Raw Blame History

created path project tags type
2026-05-22 Sources/Homelab herbys-dev-setup
proxmox
pbs
session-notes

Outcome

Set up a complete off-site backup pipeline from Proxmox VE through PBS LXC to NAS cold storage, plus validated the full round-trip restore on day one. Both local and DR restore paths tested with a running VM clone.

Final architecture:

  • PVE runs daily backup jobs to local PBS datastore main at /pbsstore/datastore (bind-mounted into LXC 110 as /datastore)
  • Weekly systemd timer on PVE host runs rsync -a --delete /pbsstore/datastore/ /mnt/nas-pbs/pbs-mirror/ at Sun 04:00, copying chunks to NAS via NFS
  • PBS LXC never touches the NAS — sidesteps the PBS-in-LXC introspection issue entirely
  • DR procedure validated: rsync NAS mirror back to /pbsstore/datastore-test, register as PBS datastore, restore VM, boot with NIC disconnected to validate

What's working at session end:

  • PBS LXC 110 on Debian 12 (systemd 252), daemons running with correct user separation (api as root, proxy as backup)
  • Local datastore main accepting daily backups at 02:00
  • NAS NFS export mounted on PVE host at /mnt/nas-pbs, persistent fstab
  • Weekly rsync timer enabled and queued for next Sunday 04:00
  • Maintenance jobs scheduled (prune 03:00, gc 03:30, verify Sat 04:00 on main; obsolete nas-offsite-targeted jobs still exist and need cleanup since we pivoted away from a NAS-side datastore)
  • Test restore datastore rnas-offsite at /pbsstore/datastore-test kept as validated DR target (frozen at this session's state, will drift from main)

Topics Covered

PBS install gotcha — Debian 13 systemd 257. Original LXC was built on debian-13-standard template. Systemd 257 in this configuration silently failed to honor User= directives in PBS unit files. Both proxmox-backup and proxmox-backup-proxy ran as root, and the proxy refused to start. Symptoms persisted across stop/start cycles and didn't respond to Type=simple override. Resolution: destroyed CT 110, rebuilt on debian-12-standard (systemd 252). Same VMID, same hostname, same MAC for DHCP consistency. Daemons now run with correct user separation.

Bind-mount approach for NFS datastore (rejected). Mounted NFS on PVE host, bind-mounted into LXC at /mnt/nas-backup. datastore create succeeded and chunk store populated, but PBS API calls failed with misleading "unable to open chunk store" error. Traced via proxy logs to find_mounted_device returning ENOENT — PBS's filesystem introspection can't traverse PVE bind mounts of NFS shares from inside the LXC namespace.

Direct NFS in LXC (also rejected). Pivoted to mounting NFS directly inside the LXC. Required features: nesting=1,mount=nfs on the container. Initial mount blocked by no_root_squash + UID mismatch — directory was owned by trucktrav-admin:media UID 1000/1001, but PBS runs as backup UID 34. Resolved by letting PBS chown the path via no_root_squash. Mount worked, but PBS still failed with the same find_mounted_device ENOENT. Confirms the issue is PBS-in-LXC mount introspection, not the mount method.

SMB experiment (also rejected). Hypothesis: SMB mounts register differently in /proc/self/mountinfo and might pass PBS's introspection check. Set up [pbs-backup] Samba share on NAS, mounted in LXC with uid=34,gid=34 (sidesteps NFS's UID-on-the-wire identity problem entirely). Initial mount failed with vfs_ChDir Permission denied because PBS had previously chowned the directory to backup:backup via no_root_squash, blocking the trucktrav-admin SMB user. Fixed by chowning back to trucktrav-admin:media. Started a datastore create over SMB to test — abandoned before completion due to extreme slowness on cold spinners (65k mkdir over SMB).

rsync from PVE host (the actual answer). PBS's chunk store at /pbsstore/datastore/ is just a directory full of content-addressed files. rsync doesn't care that PBS thinks it owns them — it just copies bytes. PVE host already has NFS mount to NAS at /mnt/nas-pbs. Cron/systemd timer runs rsync -a --delete between them. PBS never sees the NAS path; the LXC introspection issue is sidestepped entirely. Implementation: systemd .service and .timer units, weekly Sunday 04:00. Initial bulk sync took hours; subsequent delta-transfers will be minutes.

End-to-end restore validation. Tested both local and DR restore paths with a running VM clone. Created a test datastore at /pbsstore/datastore-test (PVE bind-mounted into LXC as /datastore-test), rsynced NAS mirror back to it, registered as PBS datastore, restored HerbyDev VM 100 to a throwaway VMID. Booted clone with NIC disconnected (live VM has static IP 10.0.21.207 baked into cloud-init netplan, would conflict on a real network). Verified services, disk usage, no failed units. Restore valid for both local and NAS-rsync-mirror paths.

Key Learnings

  • Debian 12 is the supported PBS base, not Debian 13. Systemd 257 in D13 silently breaks PBS unit User= directives in LXC. Always match the base OS to PBS's officially supported version (bookworm = D12).

  • PBS-in-LXC + network filesystem datastore is fundamentally broken. PBS's find_mounted_device introspection fails for NFS, SMB, and bind-mounted-NFS inside privileged LXCs — regardless of mount method. Forum consensus is to use a full VM for PBS if NFS datastores are required. We chose the better alternative for our use case: rsync the local datastore externally.

  • rsync of the chunk store directly is a clean alternative to PBS sync jobs. PBS chunks are content-addressed immutable files, ideal for rsync delta-transfer. Avoids PBS API complexity, sidesteps LXC introspection issues, works with any destination protocol. Tradeoff: DR restore requires a copy-back step before chunks are usable. Acceptable for off-site insurance scenarios.

  • find_mounted_device errors surface as wildly misleading "no such file" messages. PBS reports the error against .chunks/ even though the actual failure is the mount introspection. Always check journalctl -u proxmox-backup-proxy for the underlying ENOENT on the mount path itself — that's the real signal.

  • Don't recreate the NAS-side directory while NFS clients are mounted. Doing so spawns stale handle cascades that take significant remount work to clear. Always: unmount clients → modify NAS → remount clients.

  • exportfs -rav is mandatory after any /etc/exports edit. Verify with exportfs -v — the file and the runtime state can drift, leading to confusing "it worked before" failures.

  • mergerfs NFS exports require fsid=. FUSE filesystems don't get auto-assigned filesystem IDs, so the export needs an explicit fsid to be stable across restarts.

  • The backup user permission rabbit hole was a red herring. PBS api daemon runs as root and handles chunk store I/O. Diagnosing whether the backup system user could read the chunk store consumed real session time but wasn't the blocker.

  • Test the restore path on day one. A backup system you've never restored from is hope, not insurance. The clone-and-boot validation took an hour and converts the entire pipeline from "should work" to "demonstrably works."

  • Cloud-init netplan injects the live VM's static IP into restored clones. Booting a clone with NIC connected = ARP conflict with the running original. Disconnect NIC before boot, or edit netplan in the clone before bringing up networking. DR documentation should call this out.

  • PVE SPICE proxy hostname needs DNS or sed-rewrite. The .vv file PVE generates uses the datacenter hostname (pve.herbylab.dev) which doesn't resolve from the tower yet. Workaround: sed -i 's|pve.herbylab.dev|10.0.21.188|' file.vv before launching remote-viewer. Real fix is Knot DNS adding an A record for pve.herbylab.dev.

  • Multiple datastore create cycles on cold spinners burn hours. Each create writes 65,536 shard subdirectories. Once you've committed to rebuild, don't do partial state changes — they each cost a full chunkstore init.

  • NFS is faster than SMB for many-small-files workloads, but the user-mapping pain is real. For Linux-to-Linux with stable UIDs and trustworthy auth, NFS wins. For mixed environments or where UID drift is a concern, SMB's user-authenticated model with uid=/gid= mount options sidesteps the entire identity problem.

Follow-ons

  • Remove the obsolete pj-nas, vj-nas prune/verify jobs and nas-offsite gc schedule from PBS — they reference a datastore that no longer exists since the pivot to rsync
  • Decide whether to keep pbs-nas-test storage and rnas-offsite datastore + /pbsstore/datastore-test as a frozen DR validation point, or tear down (currently kept)
  • Add A record for pve.herbylab.dev → 10.0.21.188 in Knot DNS so SPICE files stop needing sed-rewrite
  • Fix the inconsistent labeling — PVE shows VM 100 as "HerbyDev" but OS hostname is herbys-dev. Pick one and align both.
  • Cleanup leftover artifacts on NAS: the abandoned chunk store created during SMB experiment may still have orphan dirs at /mnt/media-cold/backups/pbs outside the pbs-mirror/ subdirectory
  • Test the weekly rsync timer next Sunday — confirm Sun 2026-05-24 04:00 fires, completes, and produces a clean delta transfer (should be minutes, not hours)
  • Consider a --dry-run test rsync occasionally to spot drift between source and mirror — would catch any cases where chunks change without timestamps updating
  • Document the DR runbook somewhere visible (vault page or pinned README) so the recovery procedure isn't only in this session note
  • Long-term: consider whether to migrate PBS to a full VM if the use case ever requires native NFS datastore support (e.g., for PBS sync jobs to another PBS instance over the network)