--- created: '2026-07-18' path: Sources/Dev project: pbs-ig-autoreply-prod-cutover tags: - pbs - n8n - automation - production - mysql - docker type: project-plan --- Move the proven staging n8n IG comment→auto-reply pipeline to the **production Linode** and go live on the real PBS Instagram account. The swap happens by **exporting the staging workflows and importing them onto the production Linode** (or UI copy-paste), then flipping every staging-specific config value to prod. The logic is already proven on staging (test board #1–#5, idempotency, hold/flush, override HMAC, error handler all green) — **the risk lives entirely in the config swap**, so treat this as a checklist, not a code change. ## Goal Real Instagram traffic flowing through the prod pipeline on the real PBS account, with: a passing controlled smoke test, observability (error→GChat) live so token/channel failures are never silent, and a proven rollback path. Done = live + verified + reversible. ## Locked Decisions - **Move method:** export the 9 workflows from staging → **import onto the prod Linode**. Prefer **file import** (`n8n import:workflow`, ID-preserving) over UI copy-paste. If copy-paste is used, the cross-reference re-point step (Phase 3.4) becomes mandatory. - **Everything imported stays INACTIVE** until the controlled smoke test passes. - **Credentials are recreated fresh on prod** with prod values — n8n exports never carry credential values, only ID references. - **Canary-first go/no-go:** SendPBSChat + Central Error Handler are activated and proven to deliver to prod GChat *before* the pipeline opens to real traffic. - The cutover is a config checklist over proven logic — do not "improve" workflow logic during cutover. ## Open Items - [ ] Confirm the prod DB is pbs-api **migration-managed** the same way staging is (drives how Phase 1 schema is applied; also folds in the owed `seen_comments` migration). - [ ] Decide: is observability (durable `error_log` sink + GChat canary) a **hard Phase-0 pre-req** (recommended — the incident that started this whole thread was a silent token failure) or a fast-follow? - [ ] Confirm move method (file import vs copy-paste). Copy-paste ⇒ Phase 3.4 is mandatory. - [ ] Fill the prod value-map (Notes) before starting. ## Build Phases ### Phase 0 — Pre-flight (before touching prod) - [ ] **Freeze staging** — stop editing; you need a stable export source. - [ ] Confirm test board green (it is). - [ ] **Run the silent-drift backstop** — diff handler nodes vs the `original/` archive; last confidence pass before real traffic. - [ ] Observability decision (Open Items) — get error→GChat + at least the `error_log` sink in place if going that way. - [ ] **Snapshot both boxes** (Linode backup / the pve `lovebug@pve!snapshot` token). Rollback insurance. - [ ] Inventory the 9 workflows to move: `insta-webhook-new`, `Sub-instra-reply-handle-new`, `instra-reel-cache-new`, `SendPBSChat_New`, `PBS: Flush Held Comments`, `PBS: Held Comments TTL Sweep`, `PBS: Seen-Comments Sweep`, `PBS: Override Reply (force DM)`, `PBS: Central Error Handler`. Leave `My workflow 3` and any `-bk` behind. ### Phase 1 — Prod infra prep (prod Linode, nothing live) - [ ] Confirm prod stack up: n8n, pbs-api, MySQL, Traefik, real GChat space. - [ ] **Prod DB schema** — apply migrations so prod has everything staging accreted: `seen_comments`, held-comments table + TTL columns, `platform_posts.title`, the `projects` columns, etc. Via pbs-api migrations, **not** ad-hoc CREATE. - [ ] **Prod n8n container env:** `OVERRIDE_WEBHOOK_SECRET=`, `NODE_FUNCTION_ALLOW_BUILTIN=crypto`. Recreate container. - [ ] **Create prod credentials** (prod values), naming them identically to staging for fast re-mapping: IG bearer (real account), FB page token (if used), `MySQL account` (prod DB), `x-api-key` (prod `PBS_API_KEY`), `gchat_HMAC` (matches the env secret), GChat webhook. - [ ] **Populate prod reel↔recipe mappings** — auto-reply only fires on mapped reels. ### Phase 2 — Move the workflows - [ ] **Export** the 9 from staging (`n8n export:workflow --all --separate` or per-id; or via API). Keep the JSON. - [ ] **Import to prod, ID-preserving** (`n8n import:workflow --separate`) — keeps all `executeWorkflow` refs + `errorWorkflow` settings intact. - [ ] Sanity-check all 9 landed. **Leave everything INACTIVE.** ### Phase 3 — Config swap (staging→prod flips) - [ ] **3.1 Re-map credentials** on every node to the prod credentials (walk each workflow's HTTP/MySQL/Crypto/Bearer nodes). - [ ] **3.2 Self-filter identity** — `insta-webhook-new` → `Check if we sent`: change to the **real PBS business account** username + IG id. *(Highest-consequence single flip — miss it and the bot self-replies or filters the wrong account.)* - [ ] **3.3 Meta webhook** — subscribe the app to `comments` on the **real IG account**; point the callback at the prod n8n webhook URL; set the prod verify token; confirm the GET handshake returns the challenge. - [ ] **3.4 (copy-paste path only)** re-point every `executeWorkflow` node to the prod workflow IDs, and set every workflow's `errorWorkflow` → prod Central Error Handler id. - [ ] **3.5 Scrub staging leftovers** — grep the workflows for `pbs-staging`, `staging-test`, hardcoded staging IDs/URLs; replace hub links + pbs-api held-comments endpoint + GChat with prod. ### Phase 4 — Controlled activation + smoke test (canary-first, no real traffic) - [ ] **4.1** Activate `SendPBSChat_New` + `PBS: Central Error Handler` first. Fire a deliberate test error (throwaway erroring workflow) → confirm the ⛔ card lands in **prod** GChat. - [ ] **4.2** Activate the pipeline + subs + sweeps. - [ ] **4.3** From a **test commenter account**, comment (with keyword) on a **mapped** prod reel → verify end-to-end: resolves → reply + DM land, no errors, `seen_comments` records it. - [ ] **4.4** Spot-check guards on prod: duplicate delivery no-ops (idempotency); no-auto-reply reel drops clean; unmapped comment holds → map → flush → replies; override "send anyway" with HMAC. - [ ] **4.5** Self-filter: comment from the **real business account** → dropped, zero outbound. ### Phase 5 — Go live + watch + rollback - [ ] **Go live** — real traffic flows (Meta delivering once 3.3 is done). - [ ] **Watch window** — monitor prod executions + GChat for the first few hours (canary/observer earns its keep here). - [ ] **Rollback:** deactivate the prod pipeline workflows — comments simply aren't processed (no bad replies). Staging stays intact as reference. Fix the wrong config value, re-run Phase 4, then reactivate. ## Notes ### The two gotchas that drive the method 1. **Workflow IDs.** Workflows call each other by ID (`executeWorkflow`: Call SendPBSChat_New, Call reel-cache, Force Reply; plus every `errorWorkflow` setting). File import preserves IDs → refs resolve. UI copy-paste assigns NEW IDs → manual re-point required (3.4). 2. **Credentials don't travel.** n8n exports contain no credential values — recreate on prod (Phase 1) and re-map each node (3.1). Nothing authenticates until this is done. ### Prod value-map (fill before starting) | Item | Prod value | |---|---| | Self-filter account (username / IG id) | | | IG business account id | | | IG bearer token | | | Webhook callback host | | | Meta verify token | | | MySQL DB / host | | | PBS_API_KEY (X-API-Key) | | | OVERRIDE_WEBHOOK_SECRET | | | GChat webhook | | | Hub base URL | | ### References - Staging source of truth: Trellis thread **1006** `pbs-staging-n8n-full-transfer` (full repair + build history); companion **1002** `pbs-hold-and-flush`. - Account roles (staging): `tjherby` = sending/self account (self-filtered); `@plantbasedsoutherner` = test commenter. On prod, self-filter flips to the real business account (3.2). - Owed follow-ups carried in from staging: fold `seen_comments` DDL into a pbs-api migration; the durable `error_log` sink + GChat canary observer (design discussed; independent-channel TBD).