wiki-vault/Sources/Dev/pbs-ig-autoreply-prod-cutover.md

---
created: '2026-07-18'
path: Sources/Dev
project: pbs-ig-autoreply-prod-cutover
tags:
- pbs
- n8n
- automation
- production
- mysql
- docker
type: project-plan
---

Move the proven staging n8n IG comment→auto-reply pipeline to the **production Linode** and go live on the real PBS Instagram account. The swap happens by **exporting the staging workflows and importing them onto the production Linode** (or UI copy-paste), then flipping every staging-specific config value to prod. The logic is already proven on staging (test board #1–#5, idempotency, hold/flush, override HMAC, error handler all green) — **the risk lives entirely in the config swap**, so treat this as a checklist, not a code change.

## Goal

Real Instagram traffic flowing through the prod pipeline on the real PBS account, with: a passing controlled smoke test, observability (error→GChat) live so token/channel failures are never silent, and a proven rollback path. Done = live + verified + reversible.

## Locked Decisions

- **Move method:** export the 9 workflows from staging → **import onto the prod Linode**. Prefer **file import** (`n8n import:workflow`, ID-preserving) over UI copy-paste. If copy-paste is used, the cross-reference re-point step (Phase 3.4) becomes mandatory.
- **Everything imported stays INACTIVE** until the controlled smoke test passes.
- **Credentials are recreated fresh on prod** with prod values — n8n exports never carry credential values, only ID references.
- **Canary-first go/no-go:** SendPBSChat + Central Error Handler are activated and proven to deliver to prod GChat *before* the pipeline opens to real traffic.
- The cutover is a config checklist over proven logic — do not "improve" workflow logic during cutover.

## Open Items

- [ ] Confirm the prod DB is pbs-api **migration-managed** the same way staging is (drives how Phase 1 schema is applied; also folds in the owed `seen_comments` migration).
- [ ] Decide: is observability (durable `error_log` sink + GChat canary) a **hard Phase-0 pre-req** (recommended — the incident that started this whole thread was a silent token failure) or a fast-follow?
- [ ] Confirm move method (file import vs copy-paste). Copy-paste ⇒ Phase 3.4 is mandatory.
- [ ] Fill the prod value-map (Notes) before starting.

## Build Phases

### Phase 0 — Pre-flight (before touching prod)
- [ ] **Freeze staging** — stop editing; you need a stable export source.
- [ ] Confirm test board green (it is).
- [ ] **Run the silent-drift backstop** — diff handler nodes vs the `original/` archive; last confidence pass before real traffic.
- [ ] Observability decision (Open Items) — get error→GChat + at least the `error_log` sink in place if going that way.
- [ ] **Snapshot both boxes** (Linode backup / the pve `lovebug@pve!snapshot` token). Rollback insurance.
- [ ] Inventory the 9 workflows to move: `insta-webhook-new`, `Sub-instra-reply-handle-new`, `instra-reel-cache-new`, `SendPBSChat_New`, `PBS: Flush Held Comments`, `PBS: Held Comments TTL Sweep`, `PBS: Seen-Comments Sweep`, `PBS: Override Reply (force DM)`, `PBS: Central Error Handler`. Leave `My workflow 3` and any `-bk` behind.

### Phase 1 — Prod infra prep (prod Linode, nothing live)
- [ ] Confirm prod stack up: n8n, pbs-api, MySQL, Traefik, real GChat space.
- [ ] **Prod DB schema** — apply migrations so prod has everything staging accreted: `seen_comments`, held-comments table + TTL columns, `platform_posts.title`, the `projects` columns, etc. Via pbs-api migrations, **not** ad-hoc CREATE.
- [ ] **Prod n8n container env:** `OVERRIDE_WEBHOOK_SECRET=<prod>`, `NODE_FUNCTION_ALLOW_BUILTIN=crypto`. Recreate container.
- [ ] **Create prod credentials** (prod values), naming them identically to staging for fast re-mapping: IG bearer (real account), FB page token (if used), `MySQL account` (prod DB), `x-api-key` (prod `PBS_API_KEY`), `gchat_HMAC` (matches the env secret), GChat webhook.
- [ ] **Populate prod reel↔recipe mappings** — auto-reply only fires on mapped reels.

### Phase 2 — Move the workflows
- [ ] **Export** the 9 from staging (`n8n export:workflow --all --separate` or per-id; or via API). Keep the JSON.
- [ ] **Import to prod, ID-preserving** (`n8n import:workflow --separate`) — keeps all `executeWorkflow` refs + `errorWorkflow` settings intact.
- [ ] Sanity-check all 9 landed. **Leave everything INACTIVE.**

### Phase 3 — Config swap (staging→prod flips)
- [ ] **3.1 Re-map credentials** on every node to the prod credentials (walk each workflow's HTTP/MySQL/Crypto/Bearer nodes).
- [ ] **3.2 Self-filter identity** — `insta-webhook-new` → `Check if we sent`: change to the **real PBS business account** username + IG id. *(Highest-consequence single flip — miss it and the bot self-replies or filters the wrong account.)*
- [ ] **3.3 Meta webhook** — subscribe the app to `comments` on the **real IG account**; point the callback at the prod n8n webhook URL; set the prod verify token; confirm the GET handshake returns the challenge.
- [ ] **3.4 (copy-paste path only)** re-point every `executeWorkflow` node to the prod workflow IDs, and set every workflow's `errorWorkflow` → prod Central Error Handler id.
- [ ] **3.5 Scrub staging leftovers** — grep the workflows for `pbs-staging`, `staging-test`, hardcoded staging IDs/URLs; replace hub links + pbs-api held-comments endpoint + GChat with prod.

### Phase 4 — Controlled activation + smoke test (canary-first, no real traffic)
- [ ] **4.1** Activate `SendPBSChat_New` + `PBS: Central Error Handler` first. Fire a deliberate test error (throwaway erroring workflow) → confirm the ⛔ card lands in **prod** GChat.
- [ ] **4.2** Activate the pipeline + subs + sweeps.
- [ ] **4.3** From a **test commenter account**, comment (with keyword) on a **mapped** prod reel → verify end-to-end: resolves → reply + DM land, no errors, `seen_comments` records it.
- [ ] **4.4** Spot-check guards on prod: duplicate delivery no-ops (idempotency); no-auto-reply reel drops clean; unmapped comment holds → map → flush → replies; override "send anyway" with HMAC.
- [ ] **4.5** Self-filter: comment from the **real business account** → dropped, zero outbound.

### Phase 5 — Go live + watch + rollback
- [ ] **Go live** — real traffic flows (Meta delivering once 3.3 is done).
- [ ] **Watch window** — monitor prod executions + GChat for the first few hours (canary/observer earns its keep here).
- [ ] **Rollback:** deactivate the prod pipeline workflows — comments simply aren't processed (no bad replies). Staging stays intact as reference. Fix the wrong config value, re-run Phase 4, then reactivate.

## Notes

### The two gotchas that drive the method
1. **Workflow IDs.** Workflows call each other by ID (`executeWorkflow`: Call SendPBSChat_New, Call reel-cache, Force Reply; plus every `errorWorkflow` setting). File import preserves IDs → refs resolve. UI copy-paste assigns NEW IDs → manual re-point required (3.4).
2. **Credentials don't travel.** n8n exports contain no credential values — recreate on prod (Phase 1) and re-map each node (3.1). Nothing authenticates until this is done.

### Prod value-map (fill before starting)
| Item | Prod value |
|---|---|
| Self-filter account (username / IG id) | |
| IG business account id | |
| IG bearer token | |
| Webhook callback host | |
| Meta verify token | |
| MySQL DB / host | |
| PBS_API_KEY (X-API-Key) | |
| OVERRIDE_WEBHOOK_SECRET | |
| GChat webhook | |
| Hub base URL | |

### References
- Staging source of truth: Trellis thread **1006** `pbs-staging-n8n-full-transfer` (full repair + build history); companion **1002** `pbs-hold-and-flush`.
- Account roles (staging): `tjherby` = sending/self account (self-filtered); `@plantbasedsoutherner` = test commenter. On prod, self-filter flips to the real business account (3.2).
- Owed follow-ups carried in from staging: fold `seen_comments` DDL into a pbs-api migration; the durable `error_log` sink + GChat canary observer (design discussed; independent-channel TBD).