Create pbsii-cicd-pipeline.md via n8n
This commit is contained in:
parent
ab0b298823
commit
aebff1be34
256
Tech/Projects/pbsii-cicd-pipeline.md
Normal file
256
Tech/Projects/pbsii-cicd-pipeline.md
Normal file
@ -0,0 +1,256 @@
|
|||||||
|
---
|
||||||
|
project: pbsii-cicd-pipeline
|
||||||
|
type: project-plan
|
||||||
|
status: active
|
||||||
|
path: Tech/Projects
|
||||||
|
tags:
|
||||||
|
- pbs
|
||||||
|
- pbsii
|
||||||
|
- github-actions
|
||||||
|
- ansible
|
||||||
|
- cicd
|
||||||
|
- docker
|
||||||
|
- deployment
|
||||||
|
created: 2026-04-18
|
||||||
|
updated: 2026-04-18
|
||||||
|
---
|
||||||
|
|
||||||
|
# PBSII CI/CD Pipeline
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
|
||||||
|
Automate deployment of PBSII layers using GitHub Actions to trigger Ansible
|
||||||
|
playbooks. Each layer deploys independently based on which files changed.
|
||||||
|
Start with the dashboard layer as the first pipeline, then extend to
|
||||||
|
collector and parser.
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
- PBSII is a monorepo on GitHub (private) with subdirectories:
|
||||||
|
`collector/`, `parser/`, `dashboard/`, `n8n/`
|
||||||
|
- Ansible already handles all server config (tagged) and Docker deployments
|
||||||
|
(migrating to per-container tags)
|
||||||
|
- A standalone pbs-hub deploy role exists as a reference pattern
|
||||||
|
- Each layer deploys differently: dashboard is Docker, collector is
|
||||||
|
systemd, parser is Python/venv
|
||||||
|
- n8n workflows are just version-controlled JSON — no CI/CD needed
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
```
|
||||||
|
GitHub Push (to dashboard/)
|
||||||
|
→ GitHub Actions workflow triggers
|
||||||
|
→ Checks out PBSII repo
|
||||||
|
→ Installs Ansible on runner
|
||||||
|
→ Runs Ansible playbook with dashboard tag
|
||||||
|
→ Ansible SSHs to Linode, deploys dashboard container
|
||||||
|
```
|
||||||
|
|
||||||
|
One workflow file per layer. Explicit and simple — no magic detection.
|
||||||
|
|
||||||
|
## Phase 1: SSH Deploy Key Setup
|
||||||
|
|
||||||
|
### 1a. Generate deploy key (local machine)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh-keygen -t ed25519 -C "github-actions-deploy" -f
|
||||||
|
~/.ssh/github_deploy_linode
|
||||||
|
```
|
||||||
|
|
||||||
|
- No passphrase (key lives encrypted in GitHub Secrets)
|
||||||
|
- This key is dedicated to GitHub Actions — not your personal key
|
||||||
|
|
||||||
|
### 1b. Install public key on Linode
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh-copy-id -i ~/.ssh/github_deploy_linode.pub your_user@SERVER_IP
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] Verify: `ssh -i ~/.ssh/github_deploy_linode your_user@SERVER_IP`
|
||||||
|
connects without password prompt
|
||||||
|
|
||||||
|
### 1c. Add GitHub Secrets
|
||||||
|
|
||||||
|
In the PBSII GitHub repo → Settings → Secrets and variables → Actions:
|
||||||
|
|
||||||
|
- [ ] `DEPLOY_SSH_KEY` — contents of `~/.ssh/github_deploy_linode` (the
|
||||||
|
private key)
|
||||||
|
- [ ] `DEPLOY_HOST` — your Linode server IP
|
||||||
|
- [ ] `DEPLOY_USER` — your SSH username on Linode
|
||||||
|
- [ ] `ANSIBLE_VAULT_PASSWORD` — your Ansible vault password (if using
|
||||||
|
vault; skip if not)
|
||||||
|
|
||||||
|
### 1d. Add deploy key to Ansible inventory
|
||||||
|
|
||||||
|
Update your Ansible playbook or inventory so it can accept the key path as
|
||||||
|
a variable. The GitHub Actions workflow will pass the key location at
|
||||||
|
runtime.
|
||||||
|
|
||||||
|
## Phase 2: Ansible Dashboard Role
|
||||||
|
|
||||||
|
### 2a. Create the dashboard Ansible role
|
||||||
|
|
||||||
|
This role should handle:
|
||||||
|
|
||||||
|
- [ ] Copy dashboard source files to server (or pull from repo on server)
|
||||||
|
- [ ] Build Docker image on server (or pull pre-built)
|
||||||
|
- [ ] Restart/recreate the dashboard container via Docker Compose
|
||||||
|
- [ ] Healthcheck to verify dashboard is responding
|
||||||
|
|
||||||
|
### 2b. Decision: Build on server vs build in CI?
|
||||||
|
|
||||||
|
Two options:
|
||||||
|
|
||||||
|
**Option A — Build on server (simpler, recommended for v1):**
|
||||||
|
- GitHub Actions SSHs in via Ansible
|
||||||
|
- Ansible pulls latest code on server, runs `docker compose up -d --build
|
||||||
|
dashboard`
|
||||||
|
- No Docker registry needed
|
||||||
|
|
||||||
|
**Option B — Build in CI, push to registry:**
|
||||||
|
- GitHub Actions builds the Docker image
|
||||||
|
- Pushes to GitHub Container Registry (ghcr.io) or Docker Hub
|
||||||
|
- Ansible pulls the image on server
|
||||||
|
- More moving parts, better for multi-server setups
|
||||||
|
|
||||||
|
Recommendation: Start with Option A. You're deploying to one server.
|
||||||
|
|
||||||
|
### 2c. Tag the role
|
||||||
|
|
||||||
|
Ensure the role/playbook can be called with a tag like `--tags dashboard`
|
||||||
|
so it only touches the dashboard.
|
||||||
|
|
||||||
|
## Phase 3: GitHub Actions Workflow
|
||||||
|
|
||||||
|
### 3a. Create workflow file
|
||||||
|
|
||||||
|
File: `.github/workflows/deploy-dashboard.yml`
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
name: Deploy Dashboard
|
||||||
|
|
||||||
|
on:
|
||||||
|
push:
|
||||||
|
branches: [main]
|
||||||
|
paths:
|
||||||
|
- 'dashboard/**'
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
deploy:
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
steps:
|
||||||
|
- name: Checkout code
|
||||||
|
uses: actions/checkout@v4
|
||||||
|
|
||||||
|
- name: Set up SSH key
|
||||||
|
run: |
|
||||||
|
mkdir -p ~/.ssh
|
||||||
|
echo "${{ secrets.DEPLOY_SSH_KEY }}" > ~/.ssh/deploy_key
|
||||||
|
chmod 600 ~/.ssh/deploy_key
|
||||||
|
ssh-keyscan -H ${{ secrets.DEPLOY_HOST }} >> ~/.ssh/known_hosts
|
||||||
|
|
||||||
|
- name: Install Ansible
|
||||||
|
run: pip install ansible
|
||||||
|
|
||||||
|
- name: Run Ansible playbook
|
||||||
|
run: |
|
||||||
|
ansible-playbook playbook.yml \
|
||||||
|
--tags dashboard \
|
||||||
|
--private-key ~/.ssh/deploy_key \
|
||||||
|
-u ${{ secrets.DEPLOY_USER }} \
|
||||||
|
-i "${{ secrets.DEPLOY_HOST }},"
|
||||||
|
env:
|
||||||
|
ANSIBLE_HOST_KEY_CHECKING: "false"
|
||||||
|
```
|
||||||
|
|
||||||
|
Key points:
|
||||||
|
- `paths: ['dashboard/**']` — only triggers on changes to the dashboard
|
||||||
|
directory
|
||||||
|
- The inventory is inline (`-i "HOST,"` with trailing comma) — no inventory
|
||||||
|
file needed in the PBSII repo
|
||||||
|
- Vault password step can be added if needed
|
||||||
|
|
||||||
|
### 3b. Test the workflow
|
||||||
|
|
||||||
|
- [ ] Push a small change to `dashboard/` on main
|
||||||
|
- [ ] Watch the Actions tab in GitHub
|
||||||
|
- [ ] Verify the dashboard redeploys on Linode
|
||||||
|
- [ ] Verify pushing to other directories does NOT trigger this workflow
|
||||||
|
|
||||||
|
## Phase 4: Extend to Other Layers
|
||||||
|
|
||||||
|
Once the dashboard pipeline is solid, repeat the pattern:
|
||||||
|
|
||||||
|
### 4a. Collector pipeline
|
||||||
|
|
||||||
|
File: `.github/workflows/deploy-collector.yml`
|
||||||
|
|
||||||
|
Trigger: `paths: ['collector/**']`
|
||||||
|
|
||||||
|
Ansible role needs to:
|
||||||
|
- [ ] Build the Go binary (either on server or in CI)
|
||||||
|
- [ ] Copy binary to server
|
||||||
|
- [ ] Restart the systemd service
|
||||||
|
- [ ] Verify collector is responding on its HTTP API
|
||||||
|
|
||||||
|
Decision: Go cross-compile in CI (fast, clean) vs build on server. CI
|
||||||
|
cross-compile is easy with Go and avoids needing Go installed on Linode.
|
||||||
|
|
||||||
|
### 4b. Parser pipeline
|
||||||
|
|
||||||
|
File: `.github/workflows/deploy-parser.yml`
|
||||||
|
|
||||||
|
Trigger: `paths: ['parser/**']`
|
||||||
|
|
||||||
|
Ansible role needs to:
|
||||||
|
- [ ] Copy parser source to server
|
||||||
|
- [ ] Ensure Python venv exists with correct deps (uv sync)
|
||||||
|
- [ ] No restart needed — n8n triggers it on schedule
|
||||||
|
|
||||||
|
## Open Questions
|
||||||
|
|
||||||
|
- [ ] Does the PBSII repo need its own Ansible playbook, or should it hook
|
||||||
|
into your existing main Ansible project? If separate, the playbook and
|
||||||
|
roles live in the PBSII repo. If shared, GitHub Actions needs access to the
|
||||||
|
Ansible repo too.
|
||||||
|
- [ ] Where does the dashboard Docker Compose service definition live — in
|
||||||
|
the PBSII repo or in your main compose file on the server?
|
||||||
|
- [ ] Do you want deployment notifications? (e.g., GitHub Actions → Google
|
||||||
|
Chat webhook on success/failure)
|
||||||
|
- [ ] Branch strategy: deploy only from `main`, or also from a `staging`
|
||||||
|
branch to staging server?
|
||||||
|
|
||||||
|
## Repo Structure After Implementation
|
||||||
|
|
||||||
|
```
|
||||||
|
pbsii/
|
||||||
|
├── .github/
|
||||||
|
│ └── workflows/
|
||||||
|
│ ├── deploy-dashboard.yml
|
||||||
|
│ ├── deploy-collector.yml
|
||||||
|
│ └── deploy-parser.yml
|
||||||
|
├── ansible/
|
||||||
|
│ ├── playbook.yml
|
||||||
|
│ └── roles/
|
||||||
|
│ ├── dashboard/
|
||||||
|
│ ├── collector/
|
||||||
|
│ └── parser/
|
||||||
|
├── collector/
|
||||||
|
├── parser/
|
||||||
|
├── dashboard/
|
||||||
|
├── n8n/
|
||||||
|
│ └── workflows/
|
||||||
|
└── sql/
|
||||||
|
└── schema.sql
|
||||||
|
```
|
||||||
|
|
||||||
|
## Key Principles
|
||||||
|
|
||||||
|
- Each layer deploys independently — a change to the parser never touches
|
||||||
|
the dashboard
|
||||||
|
- Ansible stays the deployment executor — GitHub Actions is just the trigger
|
||||||
|
- Build on server for v1 — move to CI builds later if needed
|
||||||
|
- Dedicated deploy key — revocable, scoped, no passphrase
|
||||||
|
- Start with dashboard, prove the pattern, then extend
|
||||||
|
|
||||||
|
...sent from Jenny & Travis
|
||||||
Loading…
Reference in New Issue
Block a user