Update pbsii-cicd-pipeline.md via n8n

Tech/Projects/pbsii-cicd-pipeline.md

@@ -15,114 +15,113 @@ created: 2026-04-18
 updated: 2026-04-18
 ---
 
-# PBSII CI/CD Pipeline
+# PBSII CI/CD Pipeline (Revised)
 
 ## Goal
 
-Automate deployment of PBSII layers using GitHub Actions to trigger Ansible
+Automate deployment of PBSII layers using GitHub Actions with a self-hosted
-playbooks. Each layer deploys independently based on which files changed.
+runner on `ustest1`. Each layer deploys independently based on which files
-Start with the dashboard layer as the first pipeline, then extend to
+changed. Branch strategy controls target environment: `main` → production,
+`staging` → staging. Start with the dashboard layer, then extend to
 collector and parser.
 
 ## Context
 
-- PBSII is a monorepo on GitHub (private) with subdirectories:
+- PBSII is a private monorepo on GitHub with subdirectories: `collector/`,
-`collector/`, `parser/`, `dashboard/`, `n8n/`
+`parser/`, `dashboard/`, `n8n/`
-- Ansible already handles all server config (tagged) and Docker deployments
+- Ansible project (`wp-i`) handles all server config and Docker
-(migrating to per-container tags)
+deployments, lives on `ustest1`
+- Each Docker container has its own folder and `docker-compose.yml` on the
+server (e.g., `/opt/docker/dashboard/`)
+- Ansible docker role currently deploys all containers in a loop —
+migrating to per-container tags
 - A standalone pbs-hub deploy role exists as a reference pattern
-- Each layer deploys differently: dashboard is Docker, collector is
+- Each PBSII layer deploys differently: dashboard is Docker, collector is
 systemd, parser is Python/venv
-- n8n workflows are just version-controlled JSON — no CI/CD needed
+- n8n workflows are version-controlled JSON — no CI/CD needed
 
 ## Architecture
 
 ```
-GitHub Push (to dashboard/)
+GitHub Push (to dashboard/ on main)
-  → GitHub Actions workflow triggers
+  → GitHub signals self-hosted runner on ustest1
-  → Checks out PBSII repo
+  → ustest1 already has wp-i, SSH keys, Ansible — everything it needs
-  → Installs Ansible on runner
   → Runs Ansible playbook with dashboard tag
-  → Ansible SSHs to Linode, deploys dashboard container
+  → Ansible SSHs to Linode production, deploys dashboard
+  → Google Chat notification on success/failure
+
+GitHub Push (to dashboard/ on staging)
+  → Same flow, but Ansible targets staging server
 ```
 
-One workflow file per layer. Explicit and simple — no magic detection.
+## Why Self-Hosted Runner
 
-## Phase 1: SSH Deploy Key Setup
+- `ustest1` (Proxmox homelab) already has `wp-i`, SSH keys to Linode, and
+Ansible installed
+- No secrets to copy into GitHub cloud runners
+- No cloning the `wp-i` repo onto temporary VMs
+- No rebuilding Ansible environment on every run
+- Solves the staging drift problem — staging deployments go through the
+same Ansible path as production
 
-### 1a. Generate deploy key (local machine)
+## Phase 1: Self-Hosted Runner Setup
 
-```bash
+### 1a. Install GitHub Actions runner on ustest1
-ssh-keygen -t ed25519 -C "github-actions-deploy" -f
-~/.ssh/github_deploy_linode
-```
 
-- No passphrase (key lives encrypted in GitHub Secrets)
+In the PBSII GitHub repo → Settings → Actions → Runners → New self-hosted
-- This key is dedicated to GitHub Actions — not your personal key
+runner:
 
-### 1b. Install public key on Linode
+- [ ] Follow GitHub's instructions to download and configure the runner
+agent on `ustest1`
+- [ ] Install as a systemd service so it starts on boot
+- [ ] Verify runner shows as "Idle" in GitHub repo settings
 
-```bash
+### 1b. Runner user permissions
-ssh-copy-id -i ~/.ssh/github_deploy_linode.pub your_user@SERVER_IP
-```
 
-- [ ] Verify: `ssh -i ~/.ssh/github_deploy_linode your_user@SERVER_IP`
+- [ ] The runner agent runs as a user on `ustest1` — that user needs SSH
-connects without password prompt
+access to Linode (production + staging)
+- [ ] Ensure that user has access to `wp-i` on `ustest1`
+- [ ] Ensure Ansible vault password is accessible (file or environment
+variable)
 
-### 1c. Add GitHub Secrets
+### 1c. Label the runner
 
-In the PBSII GitHub repo → Settings → Secrets and variables → Actions:
+Add labels to the runner for targeting:
 
-- [ ] `DEPLOY_SSH_KEY` — contents of `~/.ssh/github_deploy_linode` (the
+- [ ] `self-hosted` (default)
-private key)
+- [ ] `ustest1` (custom, for clarity)
-- [ ] `DEPLOY_HOST` — your Linode server IP
-- [ ] `DEPLOY_USER` — your SSH username on Linode
-- [ ] `ANSIBLE_VAULT_PASSWORD` — your Ansible vault password (if using
-vault; skip if not)
-
-### 1d. Add deploy key to Ansible inventory
-
-Update your Ansible playbook or inventory so it can accept the key path as
-a variable. The GitHub Actions workflow will pass the key location at
-runtime.
 
 ## Phase 2: Ansible Dashboard Role
 
-### 2a. Create the dashboard Ansible role
+### 2a. Create the dashboard Ansible role in wp-i
 
-This role should handle:
+Use the existing pbs-hub deploy role as a reference pattern. The role
+should:
 
-- [ ] Copy dashboard source files to server (or pull from repo on server)
+- [ ] Copy dashboard source files to server at `/opt/docker/dashboard/`
-- [ ] Build Docker image on server (or pull pre-built)
+- [ ] Copy `docker-compose.yml` and `Dockerfile` to the server
-- [ ] Restart/recreate the dashboard container via Docker Compose
+- [ ] Run `docker compose up -d --build` in the dashboard folder
 - [ ] Healthcheck to verify dashboard is responding
 
-### 2b. Decision: Build on server vs build in CI?
+### 2b. Tag the role
 
-Two options:
+- [ ] Ensure the role can be called with `--tags dashboard`
+- [ ] Verify it only touches the dashboard container, nothing else
 
-**Option A — Build on server (simpler, recommended for v1):**
+### 2c. Environment targeting
-- GitHub Actions SSHs in via Ansible
-- Ansible pulls latest code on server, runs `docker compose up -d --build
-dashboard`
-- No Docker registry needed
 
-**Option B — Build in CI, push to registry:**
+The playbook needs to accept a variable (e.g., `target_env`) that
-- GitHub Actions builds the Docker image
+determines whether it deploys to production or staging:
-- Pushes to GitHub Container Registry (ghcr.io) or Docker Hub
-- Ansible pulls the image on server
-- More moving parts, better for multi-server setups
 
-Recommendation: Start with Option A. You're deploying to one server.
+- [ ] `target_env: production` → production Linode IP, production compose
+overrides
+- [ ] `target_env: staging` → staging Linode IP, staging compose overrides
 
-### 2c. Tag the role
+This keeps the same role for both environments — only the target changes.
 
-Ensure the role/playbook can be called with a tag like `--tags dashboard`
+## Phase 3: GitHub Actions Workflow (Dashboard)
-so it only touches the dashboard.
 
-## Phase 3: GitHub Actions Workflow
+### 3a. Production workflow
-
-### 3a. Create workflow file
 
 File: `.github/workflows/deploy-dashboard.yml`
 
@@ -137,49 +136,61 @@ on:
 
 jobs:
   deploy:
-    runs-on: ubuntu-latest
+    runs-on: [self-hosted, ustest1]
     steps:
-      - name: Checkout code
+      - name: Checkout PBSII
         uses: actions/checkout@v4
 
-      - name: Set up SSH key
-        run: |
-          mkdir -p ~/.ssh
-          echo "${{ secrets.DEPLOY_SSH_KEY }}" > ~/.ssh/deploy_key
-          chmod 600 ~/.ssh/deploy_key
-          ssh-keyscan -H ${{ secrets.DEPLOY_HOST }} >> ~/.ssh/known_hosts
-
-      - name: Install Ansible
-        run: pip install ansible
-
       - name: Run Ansible playbook
         run: |
+          cd /path/to/wp-i
           ansible-playbook playbook.yml \
             --tags dashboard \
-            --private-key ~/.ssh/deploy_key \
+            -e target_env=production
-            -u ${{ secrets.DEPLOY_USER }} \
+
-            -i "${{ secrets.DEPLOY_HOST }},"
+      - name: Notify Google Chat (success)
+        if: success()
+        run: |
+          curl -X POST "$GOOGLE_CHAT_WEBHOOK" \
+            -H 'Content-Type: application/json' \
+            -d '{"text": "✅ Dashboard deployed to production (commit: ${{
+github.sha }})" }'
+
+      - name: Notify Google Chat (failure)
+        if: failure()
+        run: |
+          curl -X POST "$GOOGLE_CHAT_WEBHOOK" \
+            -H 'Content-Type: application/json' \
+            -d '{"text": "❌ Dashboard deploy FAILED (commit: ${{ github.sha
+}})" }'
     env:
-          ANSIBLE_HOST_KEY_CHECKING: "false"
+      GOOGLE_CHAT_WEBHOOK: ${{ secrets.GOOGLE_CHAT_WEBHOOK }}
 ```
 
-Key points:
+### 3b. Staging workflow
-- `paths: ['dashboard/**']` — only triggers on changes to the dashboard
-directory
-- The inventory is inline (`-i "HOST,"` with trailing comma) — no inventory
-file needed in the PBSII repo
-- Vault password step can be added if needed
 
-### 3b. Test the workflow
+File: `.github/workflows/deploy-dashboard-staging.yml`
 
-- [ ] Push a small change to `dashboard/` on main
+Same as above but:
+- Trigger: `branches: [staging]`
+- Ansible flag: `-e target_env=staging`
+- Notification says "staging" instead of "production"
+
+Alternatively, this could be one workflow file with a branch conditional —
+decide during implementation.
+
+### 3c. Test the workflow
+
+- [ ] Push a small change to `dashboard/` on `staging` branch first
 - [ ] Watch the Actions tab in GitHub
-- [ ] Verify the dashboard redeploys on Linode
+- [ ] Verify dashboard redeploys on staging server
+- [ ] Verify Google Chat notification arrives
 - [ ] Verify pushing to other directories does NOT trigger this workflow
+- [ ] Once staging is solid, test the `main` branch → production flow
 
 ## Phase 4: Extend to Other Layers
 
-Once the dashboard pipeline is solid, repeat the pattern:
+Once the dashboard pipeline is proven, repeat the pattern:
 
 ### 4a. Collector pipeline
 
@@ -188,13 +199,14 @@ File: `.github/workflows/deploy-collector.yml`
 Trigger: `paths: ['collector/**']`
 
 Ansible role needs to:
-- [ ] Build the Go binary (either on server or in CI)
+- [ ] Cross-compile Go binary in CI step (or build on server)
 - [ ] Copy binary to server
-- [ ] Restart the systemd service
+- [ ] Restart the systemd service (`systemctl restart pbs-collector`)
-- [ ] Verify collector is responding on its HTTP API
+- [ ] Verify collector responds on its HTTP API
 
-Decision: Go cross-compile in CI (fast, clean) vs build on server. CI
+Note: Go cross-compiles easily — building on `ustest1` for Linux/amd64
-cross-compile is easy with Go and avoids needing Go installed on Linode.
+target and shipping the binary to Linode avoids needing Go installed on
+production.
 
 ### 4b. Parser pipeline
 
@@ -207,18 +219,20 @@ Ansible role needs to:
 - [ ] Ensure Python venv exists with correct deps (uv sync)
 - [ ] No restart needed — n8n triggers it on schedule
 
-## Open Questions
+## Phase 5: Google Chat Notifications
 
-- [ ] Does the PBSII repo need its own Ansible playbook, or should it hook
+### 5a. Setup
-into your existing main Ansible project? If separate, the playbook and
+
-roles live in the PBSII repo. If shared, GitHub Actions needs access to the
+- [ ] Add `GOOGLE_CHAT_WEBHOOK` to GitHub Secrets (can use same webhook as
-Ansible repo too.
+existing n8n alerts, or create a dedicated one for deployments)
-- [ ] Where does the dashboard Docker Compose service definition live — in
+- [ ] Decide notification format — commit hash, branch, layer name,
-the PBSII repo or in your main compose file on the server?
+success/failure
-- [ ] Do you want deployment notifications? (e.g., GitHub Actions → Google
+
-Chat webhook on success/failure)
+### 5b. Notification content
-- [ ] Branch strategy: deploy only from `main`, or also from a `staging`
+
-branch to staging server?
+Success: `✅ [layer] deployed to [env] — commit abc1234`
+Failure: `❌ [layer] deploy FAILED on [env] — commit abc1234 — check Actions
+tab`
 
 ## Repo Structure After Implementation
 
@@ -227,30 +241,79 @@ pbsii/
 ├── .github/
 │   └── workflows/
 │       ├── deploy-dashboard.yml
+│       ├── deploy-dashboard-staging.yml
 │       ├── deploy-collector.yml
-│       └── deploy-parser.yml
+│       ├── deploy-collector-staging.yml
-├── ansible/
+│       ├── deploy-parser.yml
-│   ├── playbook.yml
+│       └── deploy-parser-staging.yml
-│   └── roles/
-│       ├── dashboard/
-│       ├── collector/
-│       └── parser/
 ├── collector/
+│   ├── README.md
+│   ├── go.mod
+│   ├── main.go
+│   ├── Makefile
+│   └── pbs-collector.service
 ├── parser/
+│   ├── README.md
+│   ├── pyproject.toml
+│   └── parser.py
 ├── dashboard/
+│   ├── README.md
+│   ├── pyproject.toml
+│   ├── Dockerfile
+│   ├── docker-compose.yml
+│   └── app.py
 ├── n8n/
 │   └── workflows/
-└── sql/
+│       └── *.json
-    └── schema.sql
+├── sql/
+│   └── schema.sql
+└── README.md
 ```
 
+Ansible roles live in `wp-i`, not in this repo:
+
+```
+wp-i/
+└── roles/
+    ├── dashboard/
+    ├── collector/
+    └── parser/
+```
+
+## GitHub Secrets Required
+
+- [ ] `GOOGLE_CHAT_WEBHOOK` — webhook URL for deployment notifications
+- [ ] `ANSIBLE_VAULT_PASSWORD` — if vault is needed for deploy (may not be
+needed if runner user has access to vault password file on `ustest1`)
+
+Note: SSH keys and Ansible inventory are already on `ustest1` — no need to
+store them in GitHub.
+
+## Open Questions
+
+- [ ] What is the path to `wp-i` on `ustest1`? Needed for the workflow `cd`
+step.
+- [ ] Which user does the runner agent run as on `ustest1`? That user needs
+SSH + Ansible access.
+- [ ] Does `ustest1` currently have SSH keys to both production and staging
+Linode? If not, set up during Phase 1.
+- [ ] One workflow file per layer per environment (6 files), or one per
+layer with branch conditionals (3 files)? Start simple, refactor if it gets
+noisy.
+- [ ] Google Chat: dedicated deployment webhook or reuse existing?
+
 ## Key Principles
 
 - Each layer deploys independently — a change to the parser never touches
 the dashboard
 - Ansible stays the deployment executor — GitHub Actions is just the trigger
+- Self-hosted runner on `ustest1` — no secrets in the cloud, no rebuilding
+environments
+- Branch strategy solves staging drift — `staging` branch → staging server,
+`main` → production
 - Build on server for v1 — move to CI builds later if needed
-- Dedicated deploy key — revocable, scoped, no passphrase
+- Google Chat notifications on every deploy — success and failure
 - Start with dashboard, prove the pattern, then extend
+- Test on staging first, always
 
 ...sent from Jenny & Travis