From 1d352fe606945ca9e4234d1b7f01342c10bdca83 Mon Sep 17 00:00:00 2001 From: herbygitea Date: Sun, 19 Apr 2026 03:49:54 +0000 Subject: [PATCH] Update pbsii-cicd-pipeline.md via n8n --- Tech/Projects/pbsii-cicd-pipeline.md | 305 ++++++++++++++++----------- 1 file changed, 184 insertions(+), 121 deletions(-) diff --git a/Tech/Projects/pbsii-cicd-pipeline.md b/Tech/Projects/pbsii-cicd-pipeline.md index 12cfa0c..cc51649 100644 --- a/Tech/Projects/pbsii-cicd-pipeline.md +++ b/Tech/Projects/pbsii-cicd-pipeline.md @@ -15,114 +15,113 @@ created: 2026-04-18 updated: 2026-04-18 --- -# PBSII CI/CD Pipeline +# PBSII CI/CD Pipeline (Revised) ## Goal -Automate deployment of PBSII layers using GitHub Actions to trigger Ansible -playbooks. Each layer deploys independently based on which files changed. -Start with the dashboard layer as the first pipeline, then extend to +Automate deployment of PBSII layers using GitHub Actions with a self-hosted +runner on `ustest1`. Each layer deploys independently based on which files +changed. Branch strategy controls target environment: `main` → production, +`staging` → staging. Start with the dashboard layer, then extend to collector and parser. ## Context -- PBSII is a monorepo on GitHub (private) with subdirectories: -`collector/`, `parser/`, `dashboard/`, `n8n/` -- Ansible already handles all server config (tagged) and Docker deployments -(migrating to per-container tags) +- PBSII is a private monorepo on GitHub with subdirectories: `collector/`, +`parser/`, `dashboard/`, `n8n/` +- Ansible project (`wp-i`) handles all server config and Docker +deployments, lives on `ustest1` +- Each Docker container has its own folder and `docker-compose.yml` on the +server (e.g., `/opt/docker/dashboard/`) +- Ansible docker role currently deploys all containers in a loop — +migrating to per-container tags - A standalone pbs-hub deploy role exists as a reference pattern -- Each layer deploys differently: dashboard is Docker, collector is +- Each PBSII layer deploys differently: dashboard is Docker, collector is systemd, parser is Python/venv -- n8n workflows are just version-controlled JSON — no CI/CD needed +- n8n workflows are version-controlled JSON — no CI/CD needed ## Architecture ``` -GitHub Push (to dashboard/) - → GitHub Actions workflow triggers - → Checks out PBSII repo - → Installs Ansible on runner +GitHub Push (to dashboard/ on main) + → GitHub signals self-hosted runner on ustest1 + → ustest1 already has wp-i, SSH keys, Ansible — everything it needs → Runs Ansible playbook with dashboard tag - → Ansible SSHs to Linode, deploys dashboard container + → Ansible SSHs to Linode production, deploys dashboard + → Google Chat notification on success/failure + +GitHub Push (to dashboard/ on staging) + → Same flow, but Ansible targets staging server ``` -One workflow file per layer. Explicit and simple — no magic detection. +## Why Self-Hosted Runner -## Phase 1: SSH Deploy Key Setup +- `ustest1` (Proxmox homelab) already has `wp-i`, SSH keys to Linode, and +Ansible installed +- No secrets to copy into GitHub cloud runners +- No cloning the `wp-i` repo onto temporary VMs +- No rebuilding Ansible environment on every run +- Solves the staging drift problem — staging deployments go through the +same Ansible path as production -### 1a. Generate deploy key (local machine) +## Phase 1: Self-Hosted Runner Setup -```bash -ssh-keygen -t ed25519 -C "github-actions-deploy" -f -~/.ssh/github_deploy_linode -``` +### 1a. Install GitHub Actions runner on ustest1 -- No passphrase (key lives encrypted in GitHub Secrets) -- This key is dedicated to GitHub Actions — not your personal key +In the PBSII GitHub repo → Settings → Actions → Runners → New self-hosted +runner: -### 1b. Install public key on Linode +- [ ] Follow GitHub's instructions to download and configure the runner +agent on `ustest1` +- [ ] Install as a systemd service so it starts on boot +- [ ] Verify runner shows as "Idle" in GitHub repo settings -```bash -ssh-copy-id -i ~/.ssh/github_deploy_linode.pub your_user@SERVER_IP -``` +### 1b. Runner user permissions -- [ ] Verify: `ssh -i ~/.ssh/github_deploy_linode your_user@SERVER_IP` -connects without password prompt +- [ ] The runner agent runs as a user on `ustest1` — that user needs SSH +access to Linode (production + staging) +- [ ] Ensure that user has access to `wp-i` on `ustest1` +- [ ] Ensure Ansible vault password is accessible (file or environment +variable) -### 1c. Add GitHub Secrets +### 1c. Label the runner -In the PBSII GitHub repo → Settings → Secrets and variables → Actions: +Add labels to the runner for targeting: -- [ ] `DEPLOY_SSH_KEY` — contents of `~/.ssh/github_deploy_linode` (the -private key) -- [ ] `DEPLOY_HOST` — your Linode server IP -- [ ] `DEPLOY_USER` — your SSH username on Linode -- [ ] `ANSIBLE_VAULT_PASSWORD` — your Ansible vault password (if using -vault; skip if not) - -### 1d. Add deploy key to Ansible inventory - -Update your Ansible playbook or inventory so it can accept the key path as -a variable. The GitHub Actions workflow will pass the key location at -runtime. +- [ ] `self-hosted` (default) +- [ ] `ustest1` (custom, for clarity) ## Phase 2: Ansible Dashboard Role -### 2a. Create the dashboard Ansible role +### 2a. Create the dashboard Ansible role in wp-i -This role should handle: +Use the existing pbs-hub deploy role as a reference pattern. The role +should: -- [ ] Copy dashboard source files to server (or pull from repo on server) -- [ ] Build Docker image on server (or pull pre-built) -- [ ] Restart/recreate the dashboard container via Docker Compose +- [ ] Copy dashboard source files to server at `/opt/docker/dashboard/` +- [ ] Copy `docker-compose.yml` and `Dockerfile` to the server +- [ ] Run `docker compose up -d --build` in the dashboard folder - [ ] Healthcheck to verify dashboard is responding -### 2b. Decision: Build on server vs build in CI? +### 2b. Tag the role -Two options: +- [ ] Ensure the role can be called with `--tags dashboard` +- [ ] Verify it only touches the dashboard container, nothing else -**Option A — Build on server (simpler, recommended for v1):** -- GitHub Actions SSHs in via Ansible -- Ansible pulls latest code on server, runs `docker compose up -d --build -dashboard` -- No Docker registry needed +### 2c. Environment targeting -**Option B — Build in CI, push to registry:** -- GitHub Actions builds the Docker image -- Pushes to GitHub Container Registry (ghcr.io) or Docker Hub -- Ansible pulls the image on server -- More moving parts, better for multi-server setups +The playbook needs to accept a variable (e.g., `target_env`) that +determines whether it deploys to production or staging: -Recommendation: Start with Option A. You're deploying to one server. +- [ ] `target_env: production` → production Linode IP, production compose +overrides +- [ ] `target_env: staging` → staging Linode IP, staging compose overrides -### 2c. Tag the role +This keeps the same role for both environments — only the target changes. -Ensure the role/playbook can be called with a tag like `--tags dashboard` -so it only touches the dashboard. +## Phase 3: GitHub Actions Workflow (Dashboard) -## Phase 3: GitHub Actions Workflow - -### 3a. Create workflow file +### 3a. Production workflow File: `.github/workflows/deploy-dashboard.yml` @@ -137,49 +136,61 @@ on: jobs: deploy: - runs-on: ubuntu-latest + runs-on: [self-hosted, ustest1] steps: - - name: Checkout code + - name: Checkout PBSII uses: actions/checkout@v4 - - name: Set up SSH key - run: | - mkdir -p ~/.ssh - echo "${{ secrets.DEPLOY_SSH_KEY }}" > ~/.ssh/deploy_key - chmod 600 ~/.ssh/deploy_key - ssh-keyscan -H ${{ secrets.DEPLOY_HOST }} >> ~/.ssh/known_hosts - - - name: Install Ansible - run: pip install ansible - - name: Run Ansible playbook run: | + cd /path/to/wp-i ansible-playbook playbook.yml \ --tags dashboard \ - --private-key ~/.ssh/deploy_key \ - -u ${{ secrets.DEPLOY_USER }} \ - -i "${{ secrets.DEPLOY_HOST }}," - env: - ANSIBLE_HOST_KEY_CHECKING: "false" + -e target_env=production + + - name: Notify Google Chat (success) + if: success() + run: | + curl -X POST "$GOOGLE_CHAT_WEBHOOK" \ + -H 'Content-Type: application/json' \ + -d '{"text": "✅ Dashboard deployed to production (commit: ${{ +github.sha }})" }' + + - name: Notify Google Chat (failure) + if: failure() + run: | + curl -X POST "$GOOGLE_CHAT_WEBHOOK" \ + -H 'Content-Type: application/json' \ + -d '{"text": "❌ Dashboard deploy FAILED (commit: ${{ github.sha +}})" }' + env: + GOOGLE_CHAT_WEBHOOK: ${{ secrets.GOOGLE_CHAT_WEBHOOK }} ``` -Key points: -- `paths: ['dashboard/**']` — only triggers on changes to the dashboard -directory -- The inventory is inline (`-i "HOST,"` with trailing comma) — no inventory -file needed in the PBSII repo -- Vault password step can be added if needed +### 3b. Staging workflow -### 3b. Test the workflow +File: `.github/workflows/deploy-dashboard-staging.yml` -- [ ] Push a small change to `dashboard/` on main +Same as above but: +- Trigger: `branches: [staging]` +- Ansible flag: `-e target_env=staging` +- Notification says "staging" instead of "production" + +Alternatively, this could be one workflow file with a branch conditional — +decide during implementation. + +### 3c. Test the workflow + +- [ ] Push a small change to `dashboard/` on `staging` branch first - [ ] Watch the Actions tab in GitHub -- [ ] Verify the dashboard redeploys on Linode +- [ ] Verify dashboard redeploys on staging server +- [ ] Verify Google Chat notification arrives - [ ] Verify pushing to other directories does NOT trigger this workflow +- [ ] Once staging is solid, test the `main` branch → production flow ## Phase 4: Extend to Other Layers -Once the dashboard pipeline is solid, repeat the pattern: +Once the dashboard pipeline is proven, repeat the pattern: ### 4a. Collector pipeline @@ -188,13 +199,14 @@ File: `.github/workflows/deploy-collector.yml` Trigger: `paths: ['collector/**']` Ansible role needs to: -- [ ] Build the Go binary (either on server or in CI) +- [ ] Cross-compile Go binary in CI step (or build on server) - [ ] Copy binary to server -- [ ] Restart the systemd service -- [ ] Verify collector is responding on its HTTP API +- [ ] Restart the systemd service (`systemctl restart pbs-collector`) +- [ ] Verify collector responds on its HTTP API -Decision: Go cross-compile in CI (fast, clean) vs build on server. CI -cross-compile is easy with Go and avoids needing Go installed on Linode. +Note: Go cross-compiles easily — building on `ustest1` for Linux/amd64 +target and shipping the binary to Linode avoids needing Go installed on +production. ### 4b. Parser pipeline @@ -207,18 +219,20 @@ Ansible role needs to: - [ ] Ensure Python venv exists with correct deps (uv sync) - [ ] No restart needed — n8n triggers it on schedule -## Open Questions +## Phase 5: Google Chat Notifications -- [ ] Does the PBSII repo need its own Ansible playbook, or should it hook -into your existing main Ansible project? If separate, the playbook and -roles live in the PBSII repo. If shared, GitHub Actions needs access to the -Ansible repo too. -- [ ] Where does the dashboard Docker Compose service definition live — in -the PBSII repo or in your main compose file on the server? -- [ ] Do you want deployment notifications? (e.g., GitHub Actions → Google -Chat webhook on success/failure) -- [ ] Branch strategy: deploy only from `main`, or also from a `staging` -branch to staging server? +### 5a. Setup + +- [ ] Add `GOOGLE_CHAT_WEBHOOK` to GitHub Secrets (can use same webhook as +existing n8n alerts, or create a dedicated one for deployments) +- [ ] Decide notification format — commit hash, branch, layer name, +success/failure + +### 5b. Notification content + +Success: `✅ [layer] deployed to [env] — commit abc1234` +Failure: `❌ [layer] deploy FAILED on [env] — commit abc1234 — check Actions +tab` ## Repo Structure After Implementation @@ -227,30 +241,79 @@ pbsii/ ├── .github/ │ └── workflows/ │ ├── deploy-dashboard.yml +│ ├── deploy-dashboard-staging.yml │ ├── deploy-collector.yml -│ └── deploy-parser.yml -├── ansible/ -│ ├── playbook.yml -│ └── roles/ -│ ├── dashboard/ -│ ├── collector/ -│ └── parser/ +│ ├── deploy-collector-staging.yml +│ ├── deploy-parser.yml +│ └── deploy-parser-staging.yml ├── collector/ +│ ├── README.md +│ ├── go.mod +│ ├── main.go +│ ├── Makefile +│ └── pbs-collector.service ├── parser/ +│ ├── README.md +│ ├── pyproject.toml +│ └── parser.py ├── dashboard/ +│ ├── README.md +│ ├── pyproject.toml +│ ├── Dockerfile +│ ├── docker-compose.yml +│ └── app.py ├── n8n/ │ └── workflows/ -└── sql/ - └── schema.sql +│ └── *.json +├── sql/ +│ └── schema.sql +└── README.md ``` +Ansible roles live in `wp-i`, not in this repo: + +``` +wp-i/ +└── roles/ + ├── dashboard/ + ├── collector/ + └── parser/ +``` + +## GitHub Secrets Required + +- [ ] `GOOGLE_CHAT_WEBHOOK` — webhook URL for deployment notifications +- [ ] `ANSIBLE_VAULT_PASSWORD` — if vault is needed for deploy (may not be +needed if runner user has access to vault password file on `ustest1`) + +Note: SSH keys and Ansible inventory are already on `ustest1` — no need to +store them in GitHub. + +## Open Questions + +- [ ] What is the path to `wp-i` on `ustest1`? Needed for the workflow `cd` +step. +- [ ] Which user does the runner agent run as on `ustest1`? That user needs +SSH + Ansible access. +- [ ] Does `ustest1` currently have SSH keys to both production and staging +Linode? If not, set up during Phase 1. +- [ ] One workflow file per layer per environment (6 files), or one per +layer with branch conditionals (3 files)? Start simple, refactor if it gets +noisy. +- [ ] Google Chat: dedicated deployment webhook or reuse existing? + ## Key Principles - Each layer deploys independently — a change to the parser never touches the dashboard - Ansible stays the deployment executor — GitHub Actions is just the trigger +- Self-hosted runner on `ustest1` — no secrets in the cloud, no rebuilding +environments +- Branch strategy solves staging drift — `staging` branch → staging server, +`main` → production - Build on server for v1 — move to CI builds later if needed -- Dedicated deploy key — revocable, scoped, no passphrase +- Google Chat notifications on every deploy — success and failure - Start with dashboard, prove the pattern, then extend +- Test on staging first, always ...sent from Jenny & Travis \ No newline at end of file