wiki-vault/Sources/Dev/2026-06-22-pbs-prod-clone-and-branch-merge-2026-06-22.md

created	path	project	tags	type
2026-06-22	Sources/Dev	pbs-membership-platform-overview	pbs membership content-hub mysql wordpress ansible docker garden wpcode	session-notes

Outcome

Continuation of the prior two sessions (pbs-membership-phases-1-2-3a-deploy 2026-06-19, pbs-content-hub-sync-2026-06-20-21 2026-06-21). Today's chain (2026-06-22) did two big things:

1. Cloned production data onto the herbylab sandbox — wordpress DB (52 MB), pbs_automation DB (153 KB), wp-content/ tarball (1.2 GB → 1.7 GB extracted). Domain rewritten plantbasedsoutherner.com → pbs.herbylab.dev via wp-cli search-replace (4751 replacements). Sandbox is now a real prod mirror: themed Astra frontend, real recipe pages with real images, real users from prod (admin = sheridanbf / user_id 7), real content-hub projects.
2. Reconciled the wordpress-install branch divergence that had been latent since the 2026-06-19 chain — feat/save-button-phase2 (the Phase 2 + WP-side garden-auth work) merged into feat/content-hub-schema-sync (the schema + deploy fixes). All in-flight work now lives on one branch.

Workers run this session (W13 → W15, plus an inline patch I ran without a worker for the post-restore schema fix):

• W13 — wipe + restore wordpress + pbs_automation DBs from prod dumps. wp-cli search-replace for domain. Redis flush. Smoke-tested.
• Inline — re-applied 01_content_hub.j2 against the restored pbs_automation DB to bring in the column ALTERs that prod's schema lacked (outline/treatment/trello_card_id/trello_last_synced_at). pbs-api /pbscontenthub/ went from 500 → 200.
• W14 — extracted wp-content/ (1.2 GB tarball, scp'd in 6.8s via tailscale). Front page went 0 → 357 KB. Surfaced two branch-divergence gaps: missing REDIS_PASSWORD + GARDEN_AUTH_SHARED_SECRET in roles/dockers/files/wordpress/.env.j2; garden-auth.php not in this branch at all (lived only on save-button-phase2). Patched .env.j2 (uncommitted, handed to Travis); restored garden-auth.php via docker-cp from the host (sandbox-only fix).
• Lovebug inline commit 4811479 — landed W14's .env.j2 fix on feat/content-hub-schema-sync.
• W15 — merged feat/save-button-phase2 → feat/content-hub-schema-sync (merge commit 34c1034), resolving 6 real conflicts per a pre-agreed rules table. Verified all 9 W14 smoke checks pass post-merge. garden-auth.php is now ansible-managed (no longer host-artifact-dependent).

Final state:

• feat/content-hub-schema-sync @ 34c1034 on gitea is the canonical integration branch — has everything from W7–W15.
• feat/save-button-phase2 preserved at 4f2d105 (merged, not deleted).
• Sandbox: prod mirror, all surfaces work, member-garden Phase 3b count card renders against the (still-seeded-empty) pbs_garden.

Also: drafted + committed clone-prod-to-sandbox.md to main (cdff2c2 on gitea) — Travis's step-by-step playbook for repeating this clone procedure. Complements (doesn't replace) the existing disaster_recovery_plan.md and local-back-up-setup.md.

Topics Covered

• mysqldump idioms for wordpress + pbs_automation — --single-transaction --quick --skip-lock-tables --routines --triggers for an InnoDB consistent-snapshot. Travis dumped both from prod (GUI for the first run; switched to MYSQL_PWD="$MYSQL_ROOT_PASSWORD" inside the container after the suggestion came up). Files dropped at /opt/backups/pbs-wp/ on the dev server (not /var/backups/pbs-prod/ like my original doc said — Travis picked a different path, I followed his lead).
• scp's two-host syntax bounces through the client — not host-to-host. Modern OpenSSH 9+ removed the -3 flag because that became the default behavior with the SFTP migration. Travis's 1.5 GB scp choked partway; documented rsync --partial --inplace as the resumable alternative. (He stuck with scp; it eventually finished. Future cloning should use rsync.)
• wp-cli search-replace — installed inline into the wordpress container (ephemeral; lost on rebuild). 4751 replacements across 24 columns, including serialized blobs. --skip-columns=guid (keep guids on prod domain per WP convention). The Astra/Freemius "uninitialized class" warnings during search-replace are benign — they're serialized plugin objects wp-cli can't rehydrate but doesn't need to.
• Schema gap after restore: prod is behind pbs-api master — prod's pbs_automation doesn't have the scene/metadata tables or the trello_*/outline/treatment columns on projects. pbs-api's create_all() added the missing tables on restart but can't ALTER projects to add columns. Re-applied 01_content_hub.j2 (the patch is information_schema-guarded + idempotent) — /pbscontenthub/ immediately recovered to 200. This is the exact pattern the Alembic-centralization plan would systematize.
• wp-content tar from inside the container vs host bind mount — Travis didn't have wp-content bind-mounted on prod, used docker exec wordpress tar -czf - -C /var/www/html wp-content streaming pattern. Cleaner than the DR plan's two-command "tar to /tmp + docker cp" pattern.
• Branch divergence is the bigger story. W12 / W14 both surfaced files that "weren't in the repo" — when verified against git log --all, they ALWAYS lived on a sibling branch. The hidden cost is real: every downstream worker has to re-discover what's missing and either re-implement or stop. W15's merge is what should have happened weeks ago.
• Duplicate-YAML-key after auto-merge is an invisible footgun. Both branches added member_garden_repo_version to inventory/group_vars/herbylab/main.yml at different line positions. Git auto-merged with no conflict (the additions were textually disjoint). YAML last-wins silently picked the older pin. The first deploy after the merge ran member-garden from feat/phase-3a-garden-skeleton instead of feat/phase-3b-saved-count. W15 caught it after deploy showed the wrong card; amended the merge commit. Future merges that touch top-level YAML keys need a sanity check like yq or a key-count grep -c "^key_name:" post-merge.

Key Learnings

• W13's "schema lags pbs-api" was the inverse of the content-hub-sync chain. The 2026-06-21 chain fixed pbs-video-manager → sandbox drift; today's restore re-introduced the gap (prod schema → sandbox). Same pattern, opposite direction. The fix path was clean because we already had 01_content_hub.j2 written — re-apply against the restored DB and the columns are back.
• wp-cli is the only safe tool for WordPress URL rewriting at scale. The DR plan's sed -i 's/old/new/g' backup.sql works for non-length-changing rewrites but corrupts PHP-serialized option_values when lengths differ (which plantbasedsoutherner.com → pbs.herbylab.dev does). 4751 replacements, several inside serialized Yoast settings + Wordfence logs — wp-cli handled them faithfully; sed would have produced subtle data-corruption that wouldn't surface until specific admin pages broke. Worth normalizing.
• W14's "garden-auth.php not in the repo" was technically correct on the active branch but factually wrong overall. The file was on feat/save-button-phase2. This is the SECOND time this pattern bit a downstream worker (W12 was the first, with the env-template + Traefik labels). Pre-spawn check: git log --all --oneline -- <suspected_file> before assuming a file is missing.
• Sandbox-throwaway latitude shrinks the cost of mistakes considerably. Drop pbs_automation, drop the wp-content tree, force-rebuild images — all routine when nothing on the sandbox is worth preserving. This session ran 3 destructive operations on the sandbox (DB drops + wp-content overwrite) with zero hand-wringing because of the framing.
• container_compose_build: always is a missing convention. Every container that builds from a clone needs it; without it, docker_compose_v2 up -d reuses stale images. Member-garden has it (W12). pbs-api doesn't. The flag was added to tasks/common/deploy.yml by W12 and is available to any container — would be a small cleanup to set it for pbs-api too.
• Sheridan Fidelman = sheridanbf is the canonical admin user in prod. Useful for cross-session cookie-minting smoke tests.

Follow-ons

From this session specifically

• Set container_compose_build: always on pbs-api to match member-garden. Currently relies on manual compose build --no-cache (per W13 + W14 workarounds). Small change, prevents stale-image deploys.
• Bake wp-cli into the wordpress container image (or use a wp-cli sidecar). W13 + W14 both installed it inline (ephemeral). For repeatable clone-to-sandbox + future migration work, persistent wp-cli would save the inline install step.
• Update clone-prod-to-sandbox.md with the lessons learned today: (a) actual file path used was /opt/backups/pbs-wp/, not /var/backups/pbs-prod/; (b) the post-restore schema patch (re-apply 01_content_hub.j2) is required when prod schema lags pbs-api; (c) wp-cli install command for the WP container; (d) rsync vs scp guidance for the wp-content tarball. Travis is reviewing his cloned copy and will likely fold these in.
• YAML duplicate-key check after merges — add a step to the merge workflow (or a pre-commit hook) that runs yq or a key-count grep on any modified YAML. W15 caught the bug; we shouldn't rely on observe-and-react.
• Push feat/content-hub-schema-sync → main when ready. The branch now has all today's work; it's the canonical integration. Travis controls the merge-to-main timing (and the corresponding GitHub origin push for prod release).

Carryover from prior sessions (still open)

• Scope the Alembic-centralization vault plan (raised 2026-06-21, exhibit-A'd today again via the W13 "prod schema lags pbs-api" finding). One-time cost: baseline migration from current prod via alembic --autogenerate. Then pbs_automation schema flows through migrations like pbs_garden already does. Init-SQL files in wordpress-install shrink to CREATE DATABASE + CREATE USER. The 2026-06-21 session note has the full pitch; needs Travis green-light to turn into a vault doc.
• CSRF nonce on save/unsave (Phase 2 production-readiness; a-review flagged CRITICAL).
• member-garden migrations not automated.
• X-Forwarded-Proto (ProxyFix) in member-garden Flask.
• WP logout confirmation nonce (mu-plugin route for seamless logout).
• WPCode save-button browser test — now achievable because the sandbox has real recipe pages with WPRM markup. This is a paste-the-snippet-and-click manual test Travis can do directly; the sources are at wpcode-snippets/save-button/.
• Brand color + Sunnie SVG assets for save button + Garden (Jenny inputs).
• pbs-video-manager conftest.py reconciliation (Phase 8 fixtures lost in the same synthetic revert that hit models.py; W10 noted; not blocking pbs-api).
• init_db() multi-worker race in pbs-api gunicorn mode.
• pbs-video-manager GitHub push — main + gitea/main + gitea/master at 429383d. GitHub stays behind until Travis is ready for prod release.

Phase 3c candidate (next session, if Travis directs)

Saved-recipes detail page at /the-garden/saved (currently the Phase 3a stub). Lists saved recipes with thumbnails (joining pbs_garden.user_interactions + wordpress.wp_posts), save dates, unsave action, click-through to the real WP recipe. The sandbox now has REAL recipe data + REAL uploads, so Phase 3c can render against the real shape rather than fixtures.

Decisions worth keeping

• Drop + reapply over upsert for restoring third-party DBs. Trying to merge a prod dump into an existing schema with IF NOT EXISTS + INSERT IGNORE would leave partial fixtures intermixed with real data. Clean wipe → load → verify is faster and less surprising.
• MYSQL_PWD env var inside the container, never -p on the command line. The mysql client recognizes MYSQL_PWD; passes the password without exposing it in shell history, process listings, or ansible's argument logging.
• --partial --inplace --progress is the rsync triple for big-file transfers. Resume-on-failure, no temp-then-rename, real-time progress. The right tool when scp gets flaky on large transfers.
• Forward-fix and merge-preserve over history-rewrite. Same posture as last session's pbs-video-manager fix. The merge commit 34c1034 keeps both parents intact; feat/save-button-phase2 lives on as merged-but-discoverable history.
• Pre-spawn check for "missing file" claims: git log --all --oneline -- <path> before assuming. Burned three workers this week on the same lineage trap; saved memory feedback.