Update pbsii-cicd-pipeline.md via n8n
This commit is contained in:
parent
aebff1be34
commit
1d352fe606
@ -15,114 +15,113 @@ created: 2026-04-18
|
||||
updated: 2026-04-18
|
||||
---
|
||||
|
||||
# PBSII CI/CD Pipeline
|
||||
# PBSII CI/CD Pipeline (Revised)
|
||||
|
||||
## Goal
|
||||
|
||||
Automate deployment of PBSII layers using GitHub Actions to trigger Ansible
|
||||
playbooks. Each layer deploys independently based on which files changed.
|
||||
Start with the dashboard layer as the first pipeline, then extend to
|
||||
Automate deployment of PBSII layers using GitHub Actions with a self-hosted
|
||||
runner on `ustest1`. Each layer deploys independently based on which files
|
||||
changed. Branch strategy controls target environment: `main` → production,
|
||||
`staging` → staging. Start with the dashboard layer, then extend to
|
||||
collector and parser.
|
||||
|
||||
## Context
|
||||
|
||||
- PBSII is a monorepo on GitHub (private) with subdirectories:
|
||||
`collector/`, `parser/`, `dashboard/`, `n8n/`
|
||||
- Ansible already handles all server config (tagged) and Docker deployments
|
||||
(migrating to per-container tags)
|
||||
- PBSII is a private monorepo on GitHub with subdirectories: `collector/`,
|
||||
`parser/`, `dashboard/`, `n8n/`
|
||||
- Ansible project (`wp-i`) handles all server config and Docker
|
||||
deployments, lives on `ustest1`
|
||||
- Each Docker container has its own folder and `docker-compose.yml` on the
|
||||
server (e.g., `/opt/docker/dashboard/`)
|
||||
- Ansible docker role currently deploys all containers in a loop —
|
||||
migrating to per-container tags
|
||||
- A standalone pbs-hub deploy role exists as a reference pattern
|
||||
- Each layer deploys differently: dashboard is Docker, collector is
|
||||
- Each PBSII layer deploys differently: dashboard is Docker, collector is
|
||||
systemd, parser is Python/venv
|
||||
- n8n workflows are just version-controlled JSON — no CI/CD needed
|
||||
- n8n workflows are version-controlled JSON — no CI/CD needed
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
GitHub Push (to dashboard/)
|
||||
→ GitHub Actions workflow triggers
|
||||
→ Checks out PBSII repo
|
||||
→ Installs Ansible on runner
|
||||
GitHub Push (to dashboard/ on main)
|
||||
→ GitHub signals self-hosted runner on ustest1
|
||||
→ ustest1 already has wp-i, SSH keys, Ansible — everything it needs
|
||||
→ Runs Ansible playbook with dashboard tag
|
||||
→ Ansible SSHs to Linode, deploys dashboard container
|
||||
→ Ansible SSHs to Linode production, deploys dashboard
|
||||
→ Google Chat notification on success/failure
|
||||
|
||||
GitHub Push (to dashboard/ on staging)
|
||||
→ Same flow, but Ansible targets staging server
|
||||
```
|
||||
|
||||
One workflow file per layer. Explicit and simple — no magic detection.
|
||||
## Why Self-Hosted Runner
|
||||
|
||||
## Phase 1: SSH Deploy Key Setup
|
||||
- `ustest1` (Proxmox homelab) already has `wp-i`, SSH keys to Linode, and
|
||||
Ansible installed
|
||||
- No secrets to copy into GitHub cloud runners
|
||||
- No cloning the `wp-i` repo onto temporary VMs
|
||||
- No rebuilding Ansible environment on every run
|
||||
- Solves the staging drift problem — staging deployments go through the
|
||||
same Ansible path as production
|
||||
|
||||
### 1a. Generate deploy key (local machine)
|
||||
## Phase 1: Self-Hosted Runner Setup
|
||||
|
||||
```bash
|
||||
ssh-keygen -t ed25519 -C "github-actions-deploy" -f
|
||||
~/.ssh/github_deploy_linode
|
||||
```
|
||||
### 1a. Install GitHub Actions runner on ustest1
|
||||
|
||||
- No passphrase (key lives encrypted in GitHub Secrets)
|
||||
- This key is dedicated to GitHub Actions — not your personal key
|
||||
In the PBSII GitHub repo → Settings → Actions → Runners → New self-hosted
|
||||
runner:
|
||||
|
||||
### 1b. Install public key on Linode
|
||||
- [ ] Follow GitHub's instructions to download and configure the runner
|
||||
agent on `ustest1`
|
||||
- [ ] Install as a systemd service so it starts on boot
|
||||
- [ ] Verify runner shows as "Idle" in GitHub repo settings
|
||||
|
||||
```bash
|
||||
ssh-copy-id -i ~/.ssh/github_deploy_linode.pub your_user@SERVER_IP
|
||||
```
|
||||
### 1b. Runner user permissions
|
||||
|
||||
- [ ] Verify: `ssh -i ~/.ssh/github_deploy_linode your_user@SERVER_IP`
|
||||
connects without password prompt
|
||||
- [ ] The runner agent runs as a user on `ustest1` — that user needs SSH
|
||||
access to Linode (production + staging)
|
||||
- [ ] Ensure that user has access to `wp-i` on `ustest1`
|
||||
- [ ] Ensure Ansible vault password is accessible (file or environment
|
||||
variable)
|
||||
|
||||
### 1c. Add GitHub Secrets
|
||||
### 1c. Label the runner
|
||||
|
||||
In the PBSII GitHub repo → Settings → Secrets and variables → Actions:
|
||||
Add labels to the runner for targeting:
|
||||
|
||||
- [ ] `DEPLOY_SSH_KEY` — contents of `~/.ssh/github_deploy_linode` (the
|
||||
private key)
|
||||
- [ ] `DEPLOY_HOST` — your Linode server IP
|
||||
- [ ] `DEPLOY_USER` — your SSH username on Linode
|
||||
- [ ] `ANSIBLE_VAULT_PASSWORD` — your Ansible vault password (if using
|
||||
vault; skip if not)
|
||||
|
||||
### 1d. Add deploy key to Ansible inventory
|
||||
|
||||
Update your Ansible playbook or inventory so it can accept the key path as
|
||||
a variable. The GitHub Actions workflow will pass the key location at
|
||||
runtime.
|
||||
- [ ] `self-hosted` (default)
|
||||
- [ ] `ustest1` (custom, for clarity)
|
||||
|
||||
## Phase 2: Ansible Dashboard Role
|
||||
|
||||
### 2a. Create the dashboard Ansible role
|
||||
### 2a. Create the dashboard Ansible role in wp-i
|
||||
|
||||
This role should handle:
|
||||
Use the existing pbs-hub deploy role as a reference pattern. The role
|
||||
should:
|
||||
|
||||
- [ ] Copy dashboard source files to server (or pull from repo on server)
|
||||
- [ ] Build Docker image on server (or pull pre-built)
|
||||
- [ ] Restart/recreate the dashboard container via Docker Compose
|
||||
- [ ] Copy dashboard source files to server at `/opt/docker/dashboard/`
|
||||
- [ ] Copy `docker-compose.yml` and `Dockerfile` to the server
|
||||
- [ ] Run `docker compose up -d --build` in the dashboard folder
|
||||
- [ ] Healthcheck to verify dashboard is responding
|
||||
|
||||
### 2b. Decision: Build on server vs build in CI?
|
||||
### 2b. Tag the role
|
||||
|
||||
Two options:
|
||||
- [ ] Ensure the role can be called with `--tags dashboard`
|
||||
- [ ] Verify it only touches the dashboard container, nothing else
|
||||
|
||||
**Option A — Build on server (simpler, recommended for v1):**
|
||||
- GitHub Actions SSHs in via Ansible
|
||||
- Ansible pulls latest code on server, runs `docker compose up -d --build
|
||||
dashboard`
|
||||
- No Docker registry needed
|
||||
### 2c. Environment targeting
|
||||
|
||||
**Option B — Build in CI, push to registry:**
|
||||
- GitHub Actions builds the Docker image
|
||||
- Pushes to GitHub Container Registry (ghcr.io) or Docker Hub
|
||||
- Ansible pulls the image on server
|
||||
- More moving parts, better for multi-server setups
|
||||
The playbook needs to accept a variable (e.g., `target_env`) that
|
||||
determines whether it deploys to production or staging:
|
||||
|
||||
Recommendation: Start with Option A. You're deploying to one server.
|
||||
- [ ] `target_env: production` → production Linode IP, production compose
|
||||
overrides
|
||||
- [ ] `target_env: staging` → staging Linode IP, staging compose overrides
|
||||
|
||||
### 2c. Tag the role
|
||||
This keeps the same role for both environments — only the target changes.
|
||||
|
||||
Ensure the role/playbook can be called with a tag like `--tags dashboard`
|
||||
so it only touches the dashboard.
|
||||
## Phase 3: GitHub Actions Workflow (Dashboard)
|
||||
|
||||
## Phase 3: GitHub Actions Workflow
|
||||
|
||||
### 3a. Create workflow file
|
||||
### 3a. Production workflow
|
||||
|
||||
File: `.github/workflows/deploy-dashboard.yml`
|
||||
|
||||
@ -137,49 +136,61 @@ on:
|
||||
|
||||
jobs:
|
||||
deploy:
|
||||
runs-on: ubuntu-latest
|
||||
runs-on: [self-hosted, ustest1]
|
||||
steps:
|
||||
- name: Checkout code
|
||||
- name: Checkout PBSII
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Set up SSH key
|
||||
run: |
|
||||
mkdir -p ~/.ssh
|
||||
echo "${{ secrets.DEPLOY_SSH_KEY }}" > ~/.ssh/deploy_key
|
||||
chmod 600 ~/.ssh/deploy_key
|
||||
ssh-keyscan -H ${{ secrets.DEPLOY_HOST }} >> ~/.ssh/known_hosts
|
||||
|
||||
- name: Install Ansible
|
||||
run: pip install ansible
|
||||
|
||||
- name: Run Ansible playbook
|
||||
run: |
|
||||
cd /path/to/wp-i
|
||||
ansible-playbook playbook.yml \
|
||||
--tags dashboard \
|
||||
--private-key ~/.ssh/deploy_key \
|
||||
-u ${{ secrets.DEPLOY_USER }} \
|
||||
-i "${{ secrets.DEPLOY_HOST }},"
|
||||
env:
|
||||
ANSIBLE_HOST_KEY_CHECKING: "false"
|
||||
-e target_env=production
|
||||
|
||||
- name: Notify Google Chat (success)
|
||||
if: success()
|
||||
run: |
|
||||
curl -X POST "$GOOGLE_CHAT_WEBHOOK" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"text": "✅ Dashboard deployed to production (commit: ${{
|
||||
github.sha }})" }'
|
||||
|
||||
- name: Notify Google Chat (failure)
|
||||
if: failure()
|
||||
run: |
|
||||
curl -X POST "$GOOGLE_CHAT_WEBHOOK" \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"text": "❌ Dashboard deploy FAILED (commit: ${{ github.sha
|
||||
}})" }'
|
||||
env:
|
||||
GOOGLE_CHAT_WEBHOOK: ${{ secrets.GOOGLE_CHAT_WEBHOOK }}
|
||||
```
|
||||
|
||||
Key points:
|
||||
- `paths: ['dashboard/**']` — only triggers on changes to the dashboard
|
||||
directory
|
||||
- The inventory is inline (`-i "HOST,"` with trailing comma) — no inventory
|
||||
file needed in the PBSII repo
|
||||
- Vault password step can be added if needed
|
||||
### 3b. Staging workflow
|
||||
|
||||
### 3b. Test the workflow
|
||||
File: `.github/workflows/deploy-dashboard-staging.yml`
|
||||
|
||||
- [ ] Push a small change to `dashboard/` on main
|
||||
Same as above but:
|
||||
- Trigger: `branches: [staging]`
|
||||
- Ansible flag: `-e target_env=staging`
|
||||
- Notification says "staging" instead of "production"
|
||||
|
||||
Alternatively, this could be one workflow file with a branch conditional —
|
||||
decide during implementation.
|
||||
|
||||
### 3c. Test the workflow
|
||||
|
||||
- [ ] Push a small change to `dashboard/` on `staging` branch first
|
||||
- [ ] Watch the Actions tab in GitHub
|
||||
- [ ] Verify the dashboard redeploys on Linode
|
||||
- [ ] Verify dashboard redeploys on staging server
|
||||
- [ ] Verify Google Chat notification arrives
|
||||
- [ ] Verify pushing to other directories does NOT trigger this workflow
|
||||
- [ ] Once staging is solid, test the `main` branch → production flow
|
||||
|
||||
## Phase 4: Extend to Other Layers
|
||||
|
||||
Once the dashboard pipeline is solid, repeat the pattern:
|
||||
Once the dashboard pipeline is proven, repeat the pattern:
|
||||
|
||||
### 4a. Collector pipeline
|
||||
|
||||
@ -188,13 +199,14 @@ File: `.github/workflows/deploy-collector.yml`
|
||||
Trigger: `paths: ['collector/**']`
|
||||
|
||||
Ansible role needs to:
|
||||
- [ ] Build the Go binary (either on server or in CI)
|
||||
- [ ] Cross-compile Go binary in CI step (or build on server)
|
||||
- [ ] Copy binary to server
|
||||
- [ ] Restart the systemd service
|
||||
- [ ] Verify collector is responding on its HTTP API
|
||||
- [ ] Restart the systemd service (`systemctl restart pbs-collector`)
|
||||
- [ ] Verify collector responds on its HTTP API
|
||||
|
||||
Decision: Go cross-compile in CI (fast, clean) vs build on server. CI
|
||||
cross-compile is easy with Go and avoids needing Go installed on Linode.
|
||||
Note: Go cross-compiles easily — building on `ustest1` for Linux/amd64
|
||||
target and shipping the binary to Linode avoids needing Go installed on
|
||||
production.
|
||||
|
||||
### 4b. Parser pipeline
|
||||
|
||||
@ -207,18 +219,20 @@ Ansible role needs to:
|
||||
- [ ] Ensure Python venv exists with correct deps (uv sync)
|
||||
- [ ] No restart needed — n8n triggers it on schedule
|
||||
|
||||
## Open Questions
|
||||
## Phase 5: Google Chat Notifications
|
||||
|
||||
- [ ] Does the PBSII repo need its own Ansible playbook, or should it hook
|
||||
into your existing main Ansible project? If separate, the playbook and
|
||||
roles live in the PBSII repo. If shared, GitHub Actions needs access to the
|
||||
Ansible repo too.
|
||||
- [ ] Where does the dashboard Docker Compose service definition live — in
|
||||
the PBSII repo or in your main compose file on the server?
|
||||
- [ ] Do you want deployment notifications? (e.g., GitHub Actions → Google
|
||||
Chat webhook on success/failure)
|
||||
- [ ] Branch strategy: deploy only from `main`, or also from a `staging`
|
||||
branch to staging server?
|
||||
### 5a. Setup
|
||||
|
||||
- [ ] Add `GOOGLE_CHAT_WEBHOOK` to GitHub Secrets (can use same webhook as
|
||||
existing n8n alerts, or create a dedicated one for deployments)
|
||||
- [ ] Decide notification format — commit hash, branch, layer name,
|
||||
success/failure
|
||||
|
||||
### 5b. Notification content
|
||||
|
||||
Success: `✅ [layer] deployed to [env] — commit abc1234`
|
||||
Failure: `❌ [layer] deploy FAILED on [env] — commit abc1234 — check Actions
|
||||
tab`
|
||||
|
||||
## Repo Structure After Implementation
|
||||
|
||||
@ -227,30 +241,79 @@ pbsii/
|
||||
├── .github/
|
||||
│ └── workflows/
|
||||
│ ├── deploy-dashboard.yml
|
||||
│ ├── deploy-dashboard-staging.yml
|
||||
│ ├── deploy-collector.yml
|
||||
│ └── deploy-parser.yml
|
||||
├── ansible/
|
||||
│ ├── playbook.yml
|
||||
│ └── roles/
|
||||
│ ├── dashboard/
|
||||
│ ├── collector/
|
||||
│ └── parser/
|
||||
│ ├── deploy-collector-staging.yml
|
||||
│ ├── deploy-parser.yml
|
||||
│ └── deploy-parser-staging.yml
|
||||
├── collector/
|
||||
│ ├── README.md
|
||||
│ ├── go.mod
|
||||
│ ├── main.go
|
||||
│ ├── Makefile
|
||||
│ └── pbs-collector.service
|
||||
├── parser/
|
||||
│ ├── README.md
|
||||
│ ├── pyproject.toml
|
||||
│ └── parser.py
|
||||
├── dashboard/
|
||||
│ ├── README.md
|
||||
│ ├── pyproject.toml
|
||||
│ ├── Dockerfile
|
||||
│ ├── docker-compose.yml
|
||||
│ └── app.py
|
||||
├── n8n/
|
||||
│ └── workflows/
|
||||
└── sql/
|
||||
└── schema.sql
|
||||
│ └── *.json
|
||||
├── sql/
|
||||
│ └── schema.sql
|
||||
└── README.md
|
||||
```
|
||||
|
||||
Ansible roles live in `wp-i`, not in this repo:
|
||||
|
||||
```
|
||||
wp-i/
|
||||
└── roles/
|
||||
├── dashboard/
|
||||
├── collector/
|
||||
└── parser/
|
||||
```
|
||||
|
||||
## GitHub Secrets Required
|
||||
|
||||
- [ ] `GOOGLE_CHAT_WEBHOOK` — webhook URL for deployment notifications
|
||||
- [ ] `ANSIBLE_VAULT_PASSWORD` — if vault is needed for deploy (may not be
|
||||
needed if runner user has access to vault password file on `ustest1`)
|
||||
|
||||
Note: SSH keys and Ansible inventory are already on `ustest1` — no need to
|
||||
store them in GitHub.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- [ ] What is the path to `wp-i` on `ustest1`? Needed for the workflow `cd`
|
||||
step.
|
||||
- [ ] Which user does the runner agent run as on `ustest1`? That user needs
|
||||
SSH + Ansible access.
|
||||
- [ ] Does `ustest1` currently have SSH keys to both production and staging
|
||||
Linode? If not, set up during Phase 1.
|
||||
- [ ] One workflow file per layer per environment (6 files), or one per
|
||||
layer with branch conditionals (3 files)? Start simple, refactor if it gets
|
||||
noisy.
|
||||
- [ ] Google Chat: dedicated deployment webhook or reuse existing?
|
||||
|
||||
## Key Principles
|
||||
|
||||
- Each layer deploys independently — a change to the parser never touches
|
||||
the dashboard
|
||||
- Ansible stays the deployment executor — GitHub Actions is just the trigger
|
||||
- Self-hosted runner on `ustest1` — no secrets in the cloud, no rebuilding
|
||||
environments
|
||||
- Branch strategy solves staging drift — `staging` branch → staging server,
|
||||
`main` → production
|
||||
- Build on server for v1 — move to CI builds later if needed
|
||||
- Dedicated deploy key — revocable, scoped, no passphrase
|
||||
- Google Chat notifications on every deploy — success and failure
|
||||
- Start with dashboard, prove the pattern, then extend
|
||||
- Test on staging first, always
|
||||
|
||||
...sent from Jenny & Travis
|
||||
Loading…
Reference in New Issue
Block a user