petal-power/wiki-vault
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
e799165d1b
wiki-vault/Sources/Dev/pbs-ig-autoreply-prod-cutover.md

7.9 KiB
Raw Blame History

created path project tags type
2026-07-18 Sources/Dev pbs-ig-autoreply-prod-cutover
pbs
n8n
automation
production
mysql
docker
project-plan

Move the proven staging n8n IG comment→auto-reply pipeline to the production Linode and go live on the real PBS Instagram account. The swap happens by exporting the staging workflows and importing them onto the production Linode (or UI copy-paste), then flipping every staging-specific config value to prod. The logic is already proven on staging (test board #1–#5, idempotency, hold/flush, override HMAC, error handler all green) — the risk lives entirely in the config swap, so treat this as a checklist, not a code change.

Goal

Real Instagram traffic flowing through the prod pipeline on the real PBS account, with: a passing controlled smoke test, observability (error→GChat) live so token/channel failures are never silent, and a proven rollback path. Done = live + verified + reversible.

Locked Decisions

  • Move method: export the 9 workflows from staging → import onto the prod Linode. Prefer file import (n8n import:workflow, ID-preserving) over UI copy-paste. If copy-paste is used, the cross-reference re-point step (Phase 3.4) becomes mandatory.
  • Everything imported stays INACTIVE until the controlled smoke test passes.
  • Credentials are recreated fresh on prod with prod values — n8n exports never carry credential values, only ID references.
  • Canary-first go/no-go: SendPBSChat + Central Error Handler are activated and proven to deliver to prod GChat before the pipeline opens to real traffic.
  • The cutover is a config checklist over proven logic — do not "improve" workflow logic during cutover.

Open Items

  • Confirm the prod DB is pbs-api migration-managed the same way staging is (drives how Phase 1 schema is applied; also folds in the owed seen_comments migration).
  • Decide: is observability (durable error_log sink + GChat canary) a hard Phase-0 pre-req (recommended — the incident that started this whole thread was a silent token failure) or a fast-follow?
  • Confirm move method (file import vs copy-paste). Copy-paste ⇒ Phase 3.4 is mandatory.
  • Fill the prod value-map (Notes) before starting.

Build Phases

Phase 0 — Pre-flight (before touching prod)

  • Freeze staging — stop editing; you need a stable export source.
  • Confirm test board green (it is).
  • Run the silent-drift backstop — diff handler nodes vs the original/ archive; last confidence pass before real traffic.
  • Observability decision (Open Items) — get error→GChat + at least the error_log sink in place if going that way.
  • Snapshot both boxes (Linode backup / the pve lovebug@pve!snapshot token). Rollback insurance.
  • Inventory the 9 workflows to move: insta-webhook-new, Sub-instra-reply-handle-new, instra-reel-cache-new, SendPBSChat_New, PBS: Flush Held Comments, PBS: Held Comments TTL Sweep, PBS: Seen-Comments Sweep, PBS: Override Reply (force DM), PBS: Central Error Handler. Leave My workflow 3 and any -bk behind.

Phase 1 — Prod infra prep (prod Linode, nothing live)

  • Confirm prod stack up: n8n, pbs-api, MySQL, Traefik, real GChat space.
  • Prod DB schema — apply migrations so prod has everything staging accreted: seen_comments, held-comments table + TTL columns, platform_posts.title, the projects columns, etc. Via pbs-api migrations, not ad-hoc CREATE.
  • Prod n8n container env: OVERRIDE_WEBHOOK_SECRET=<prod>, NODE_FUNCTION_ALLOW_BUILTIN=crypto. Recreate container.
  • Create prod credentials (prod values), naming them identically to staging for fast re-mapping: IG bearer (real account), FB page token (if used), MySQL account (prod DB), x-api-key (prod PBS_API_KEY), gchat_HMAC (matches the env secret), GChat webhook.
  • Populate prod reel↔recipe mappings — auto-reply only fires on mapped reels.

Phase 2 — Move the workflows

  • Export the 9 from staging (n8n export:workflow --all --separate or per-id; or via API). Keep the JSON.
  • Import to prod, ID-preserving (n8n import:workflow --separate) — keeps all executeWorkflow refs + errorWorkflow settings intact.
  • Sanity-check all 9 landed. Leave everything INACTIVE.

Phase 3 — Config swap (staging→prod flips)

  • 3.1 Re-map credentials on every node to the prod credentials (walk each workflow's HTTP/MySQL/Crypto/Bearer nodes).
  • 3.2 Self-filter identity — insta-webhook-new → Check if we sent: change to the real PBS business account username + IG id. (Highest-consequence single flip — miss it and the bot self-replies or filters the wrong account.)
  • 3.3 Meta webhook — subscribe the app to comments on the real IG account; point the callback at the prod n8n webhook URL; set the prod verify token; confirm the GET handshake returns the challenge.
  • 3.4 (copy-paste path only) re-point every executeWorkflow node to the prod workflow IDs, and set every workflow's errorWorkflow → prod Central Error Handler id.
  • 3.5 Scrub staging leftovers — grep the workflows for pbs-staging, staging-test, hardcoded staging IDs/URLs; replace hub links + pbs-api held-comments endpoint + GChat with prod.

Phase 4 — Controlled activation + smoke test (canary-first, no real traffic)

  • 4.1 Activate SendPBSChat_New + PBS: Central Error Handler first. Fire a deliberate test error (throwaway erroring workflow) → confirm the ⛔ card lands in prod GChat.
  • 4.2 Activate the pipeline + subs + sweeps.
  • 4.3 From a test commenter account, comment (with keyword) on a mapped prod reel → verify end-to-end: resolves → reply + DM land, no errors, seen_comments records it.
  • 4.4 Spot-check guards on prod: duplicate delivery no-ops (idempotency); no-auto-reply reel drops clean; unmapped comment holds → map → flush → replies; override "send anyway" with HMAC.
  • 4.5 Self-filter: comment from the real business account → dropped, zero outbound.

Phase 5 — Go live + watch + rollback

  • Go live — real traffic flows (Meta delivering once 3.3 is done).
  • Watch window — monitor prod executions + GChat for the first few hours (canary/observer earns its keep here).
  • Rollback: deactivate the prod pipeline workflows — comments simply aren't processed (no bad replies). Staging stays intact as reference. Fix the wrong config value, re-run Phase 4, then reactivate.

Notes

The two gotchas that drive the method

  1. Workflow IDs. Workflows call each other by ID (executeWorkflow: Call SendPBSChat_New, Call reel-cache, Force Reply; plus every errorWorkflow setting). File import preserves IDs → refs resolve. UI copy-paste assigns NEW IDs → manual re-point required (3.4).
  2. Credentials don't travel. n8n exports contain no credential values — recreate on prod (Phase 1) and re-map each node (3.1). Nothing authenticates until this is done.

Prod value-map (fill before starting)

Item Prod value
Self-filter account (username / IG id)
IG business account id
IG bearer token
Webhook callback host
Meta verify token
MySQL DB / host
PBS_API_KEY (X-API-Key)
OVERRIDE_WEBHOOK_SECRET
GChat webhook
Hub base URL

References

  • Staging source of truth: Trellis thread 1006 pbs-staging-n8n-full-transfer (full repair + build history); companion 1002 pbs-hold-and-flush.
  • Account roles (staging): tjherby = sending/self account (self-filtered); @plantbasedsoutherner = test commenter. On prod, self-filter flips to the real business account (3.2).
  • Owed follow-ups carried in from staging: fold seen_comments DDL into a pbs-api migration; the durable error_log sink + GChat canary observer (design discussed; independent-channel TBD).