Create pbsii-cicd-pipeline.md via n8n
This commit is contained in:
parent
ab0b298823
commit
aebff1be34
256
Tech/Projects/pbsii-cicd-pipeline.md
Normal file
256
Tech/Projects/pbsii-cicd-pipeline.md
Normal file
@ -0,0 +1,256 @@
|
||||
---
|
||||
project: pbsii-cicd-pipeline
|
||||
type: project-plan
|
||||
status: active
|
||||
path: Tech/Projects
|
||||
tags:
|
||||
- pbs
|
||||
- pbsii
|
||||
- github-actions
|
||||
- ansible
|
||||
- cicd
|
||||
- docker
|
||||
- deployment
|
||||
created: 2026-04-18
|
||||
updated: 2026-04-18
|
||||
---
|
||||
|
||||
# PBSII CI/CD Pipeline
|
||||
|
||||
## Goal
|
||||
|
||||
Automate deployment of PBSII layers using GitHub Actions to trigger Ansible
|
||||
playbooks. Each layer deploys independently based on which files changed.
|
||||
Start with the dashboard layer as the first pipeline, then extend to
|
||||
collector and parser.
|
||||
|
||||
## Context
|
||||
|
||||
- PBSII is a monorepo on GitHub (private) with subdirectories:
|
||||
`collector/`, `parser/`, `dashboard/`, `n8n/`
|
||||
- Ansible already handles all server config (tagged) and Docker deployments
|
||||
(migrating to per-container tags)
|
||||
- A standalone pbs-hub deploy role exists as a reference pattern
|
||||
- Each layer deploys differently: dashboard is Docker, collector is
|
||||
systemd, parser is Python/venv
|
||||
- n8n workflows are just version-controlled JSON — no CI/CD needed
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
GitHub Push (to dashboard/)
|
||||
→ GitHub Actions workflow triggers
|
||||
→ Checks out PBSII repo
|
||||
→ Installs Ansible on runner
|
||||
→ Runs Ansible playbook with dashboard tag
|
||||
→ Ansible SSHs to Linode, deploys dashboard container
|
||||
```
|
||||
|
||||
One workflow file per layer. Explicit and simple — no magic detection.
|
||||
|
||||
## Phase 1: SSH Deploy Key Setup
|
||||
|
||||
### 1a. Generate deploy key (local machine)
|
||||
|
||||
```bash
|
||||
ssh-keygen -t ed25519 -C "github-actions-deploy" -f
|
||||
~/.ssh/github_deploy_linode
|
||||
```
|
||||
|
||||
- No passphrase (key lives encrypted in GitHub Secrets)
|
||||
- This key is dedicated to GitHub Actions — not your personal key
|
||||
|
||||
### 1b. Install public key on Linode
|
||||
|
||||
```bash
|
||||
ssh-copy-id -i ~/.ssh/github_deploy_linode.pub your_user@SERVER_IP
|
||||
```
|
||||
|
||||
- [ ] Verify: `ssh -i ~/.ssh/github_deploy_linode your_user@SERVER_IP`
|
||||
connects without password prompt
|
||||
|
||||
### 1c. Add GitHub Secrets
|
||||
|
||||
In the PBSII GitHub repo → Settings → Secrets and variables → Actions:
|
||||
|
||||
- [ ] `DEPLOY_SSH_KEY` — contents of `~/.ssh/github_deploy_linode` (the
|
||||
private key)
|
||||
- [ ] `DEPLOY_HOST` — your Linode server IP
|
||||
- [ ] `DEPLOY_USER` — your SSH username on Linode
|
||||
- [ ] `ANSIBLE_VAULT_PASSWORD` — your Ansible vault password (if using
|
||||
vault; skip if not)
|
||||
|
||||
### 1d. Add deploy key to Ansible inventory
|
||||
|
||||
Update your Ansible playbook or inventory so it can accept the key path as
|
||||
a variable. The GitHub Actions workflow will pass the key location at
|
||||
runtime.
|
||||
|
||||
## Phase 2: Ansible Dashboard Role
|
||||
|
||||
### 2a. Create the dashboard Ansible role
|
||||
|
||||
This role should handle:
|
||||
|
||||
- [ ] Copy dashboard source files to server (or pull from repo on server)
|
||||
- [ ] Build Docker image on server (or pull pre-built)
|
||||
- [ ] Restart/recreate the dashboard container via Docker Compose
|
||||
- [ ] Healthcheck to verify dashboard is responding
|
||||
|
||||
### 2b. Decision: Build on server vs build in CI?
|
||||
|
||||
Two options:
|
||||
|
||||
**Option A — Build on server (simpler, recommended for v1):**
|
||||
- GitHub Actions SSHs in via Ansible
|
||||
- Ansible pulls latest code on server, runs `docker compose up -d --build
|
||||
dashboard`
|
||||
- No Docker registry needed
|
||||
|
||||
**Option B — Build in CI, push to registry:**
|
||||
- GitHub Actions builds the Docker image
|
||||
- Pushes to GitHub Container Registry (ghcr.io) or Docker Hub
|
||||
- Ansible pulls the image on server
|
||||
- More moving parts, better for multi-server setups
|
||||
|
||||
Recommendation: Start with Option A. You're deploying to one server.
|
||||
|
||||
### 2c. Tag the role
|
||||
|
||||
Ensure the role/playbook can be called with a tag like `--tags dashboard`
|
||||
so it only touches the dashboard.
|
||||
|
||||
## Phase 3: GitHub Actions Workflow
|
||||
|
||||
### 3a. Create workflow file
|
||||
|
||||
File: `.github/workflows/deploy-dashboard.yml`
|
||||
|
||||
```yaml
|
||||
name: Deploy Dashboard
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
paths:
|
||||
- 'dashboard/**'
|
||||
|
||||
jobs:
|
||||
deploy:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Checkout code
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Set up SSH key
|
||||
run: |
|
||||
mkdir -p ~/.ssh
|
||||
echo "${{ secrets.DEPLOY_SSH_KEY }}" > ~/.ssh/deploy_key
|
||||
chmod 600 ~/.ssh/deploy_key
|
||||
ssh-keyscan -H ${{ secrets.DEPLOY_HOST }} >> ~/.ssh/known_hosts
|
||||
|
||||
- name: Install Ansible
|
||||
run: pip install ansible
|
||||
|
||||
- name: Run Ansible playbook
|
||||
run: |
|
||||
ansible-playbook playbook.yml \
|
||||
--tags dashboard \
|
||||
--private-key ~/.ssh/deploy_key \
|
||||
-u ${{ secrets.DEPLOY_USER }} \
|
||||
-i "${{ secrets.DEPLOY_HOST }},"
|
||||
env:
|
||||
ANSIBLE_HOST_KEY_CHECKING: "false"
|
||||
```
|
||||
|
||||
Key points:
|
||||
- `paths: ['dashboard/**']` — only triggers on changes to the dashboard
|
||||
directory
|
||||
- The inventory is inline (`-i "HOST,"` with trailing comma) — no inventory
|
||||
file needed in the PBSII repo
|
||||
- Vault password step can be added if needed
|
||||
|
||||
### 3b. Test the workflow
|
||||
|
||||
- [ ] Push a small change to `dashboard/` on main
|
||||
- [ ] Watch the Actions tab in GitHub
|
||||
- [ ] Verify the dashboard redeploys on Linode
|
||||
- [ ] Verify pushing to other directories does NOT trigger this workflow
|
||||
|
||||
## Phase 4: Extend to Other Layers
|
||||
|
||||
Once the dashboard pipeline is solid, repeat the pattern:
|
||||
|
||||
### 4a. Collector pipeline
|
||||
|
||||
File: `.github/workflows/deploy-collector.yml`
|
||||
|
||||
Trigger: `paths: ['collector/**']`
|
||||
|
||||
Ansible role needs to:
|
||||
- [ ] Build the Go binary (either on server or in CI)
|
||||
- [ ] Copy binary to server
|
||||
- [ ] Restart the systemd service
|
||||
- [ ] Verify collector is responding on its HTTP API
|
||||
|
||||
Decision: Go cross-compile in CI (fast, clean) vs build on server. CI
|
||||
cross-compile is easy with Go and avoids needing Go installed on Linode.
|
||||
|
||||
### 4b. Parser pipeline
|
||||
|
||||
File: `.github/workflows/deploy-parser.yml`
|
||||
|
||||
Trigger: `paths: ['parser/**']`
|
||||
|
||||
Ansible role needs to:
|
||||
- [ ] Copy parser source to server
|
||||
- [ ] Ensure Python venv exists with correct deps (uv sync)
|
||||
- [ ] No restart needed — n8n triggers it on schedule
|
||||
|
||||
## Open Questions
|
||||
|
||||
- [ ] Does the PBSII repo need its own Ansible playbook, or should it hook
|
||||
into your existing main Ansible project? If separate, the playbook and
|
||||
roles live in the PBSII repo. If shared, GitHub Actions needs access to the
|
||||
Ansible repo too.
|
||||
- [ ] Where does the dashboard Docker Compose service definition live — in
|
||||
the PBSII repo or in your main compose file on the server?
|
||||
- [ ] Do you want deployment notifications? (e.g., GitHub Actions → Google
|
||||
Chat webhook on success/failure)
|
||||
- [ ] Branch strategy: deploy only from `main`, or also from a `staging`
|
||||
branch to staging server?
|
||||
|
||||
## Repo Structure After Implementation
|
||||
|
||||
```
|
||||
pbsii/
|
||||
├── .github/
|
||||
│ └── workflows/
|
||||
│ ├── deploy-dashboard.yml
|
||||
│ ├── deploy-collector.yml
|
||||
│ └── deploy-parser.yml
|
||||
├── ansible/
|
||||
│ ├── playbook.yml
|
||||
│ └── roles/
|
||||
│ ├── dashboard/
|
||||
│ ├── collector/
|
||||
│ └── parser/
|
||||
├── collector/
|
||||
├── parser/
|
||||
├── dashboard/
|
||||
├── n8n/
|
||||
│ └── workflows/
|
||||
└── sql/
|
||||
└── schema.sql
|
||||
```
|
||||
|
||||
## Key Principles
|
||||
|
||||
- Each layer deploys independently — a change to the parser never touches
|
||||
the dashboard
|
||||
- Ansible stays the deployment executor — GitHub Actions is just the trigger
|
||||
- Build on server for v1 — move to CI builds later if needed
|
||||
- Dedicated deploy key — revocable, scoped, no passphrase
|
||||
- Start with dashboard, prove the pattern, then extend
|
||||
|
||||
...sent from Jenny & Travis
|
||||
Loading…
Reference in New Issue
Block a user