Create pbs-infrastructure-intelligence.md via n8n
This commit is contained in:
parent
8a3bae92f4
commit
9404dd5e95
481
PBS/Tech/Projects/pbs-infrastructure-intelligence.md
Normal file
481
PBS/Tech/Projects/pbs-infrastructure-intelligence.md
Normal file
@ -0,0 +1,481 @@
|
|||||||
|
---
|
||||||
|
project: pbs-infrastructure-intelligence
|
||||||
|
type: project-plan
|
||||||
|
status: active
|
||||||
|
path: PBS/Tech/Projects
|
||||||
|
tags:
|
||||||
|
- pbs
|
||||||
|
- monitoring
|
||||||
|
- docker
|
||||||
|
- go
|
||||||
|
- python
|
||||||
|
- mysql
|
||||||
|
- n8n
|
||||||
|
- streamlit
|
||||||
|
- security
|
||||||
|
created: 2026-04-16
|
||||||
|
updated: 2026-04-16
|
||||||
|
---
|
||||||
|
|
||||||
|
# PBS Infrastructure Intelligence (PBSII)
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
|
||||||
|
Build a self-healing, intelligent monitoring system for the PBS
|
||||||
|
infrastructure. Four interconnected layers: a metrics collector, a
|
||||||
|
visualization dashboard, a threshold monitor, and an AI-powered healer.
|
||||||
|
|
||||||
|
## Problem Being Solved
|
||||||
|
|
||||||
|
- DoS attacks cause server thrashing with no early warning
|
||||||
|
- Container memory leaks go undetected until crashes occur
|
||||||
|
- Existing monitoring (Uptime Kuma) only detects hard-down states, not
|
||||||
|
degradation trends
|
||||||
|
- Manual log grepping is unsustainable as the site grows
|
||||||
|
- No visibility into what traffic reaches the server vs what Cloudflare
|
||||||
|
blocks
|
||||||
|
- Apache worker exhaustion causes memory crashes with no forensic data
|
||||||
|
|
||||||
|
## Architecture Overview
|
||||||
|
|
||||||
|
```
|
||||||
|
Layer 1: Collector (Go, systemd)
|
||||||
|
→ JSONL file (30s interval)
|
||||||
|
→ MySQL sync (every few minutes)
|
||||||
|
→ HTTP API (/metrics/current, /metrics/history, /health)
|
||||||
|
|
||||||
|
Log Parser (Python, n8n-scheduled)
|
||||||
|
→ docker logs --since
|
||||||
|
→ MySQL pbs_security_events + pbs_security_summary
|
||||||
|
|
||||||
|
Layer 2: Dashboard (Streamlit)
|
||||||
|
→ Reads MySQL
|
||||||
|
→ System metrics + security events + healer activity
|
||||||
|
|
||||||
|
Layer 3: Monitor (n8n, every 5 min)
|
||||||
|
→ Evaluates threshold rules against MySQL data
|
||||||
|
→ Fires warnings (Google Chat) or invokes healer
|
||||||
|
|
||||||
|
Layer 4: Healer (n8n + Claude API)
|
||||||
|
→ Gathers diagnostic context from all layers
|
||||||
|
→ Claude API reasons about the problem
|
||||||
|
→ Executes allowed actions or escalates
|
||||||
|
→ Logs all decisions to pbs_healer_actions
|
||||||
|
```
|
||||||
|
|
||||||
|
## Layer 1: Collector (Go Microservice)
|
||||||
|
|
||||||
|
### Purpose
|
||||||
|
|
||||||
|
Collect system and container metrics. Write to disk first, sync to MySQL
|
||||||
|
second. Serve current state via HTTP. No opinions, just data.
|
||||||
|
|
||||||
|
### Technology
|
||||||
|
|
||||||
|
- Language: Go (standard library first)
|
||||||
|
- Deployment: Single static binary, systemd service on host (not
|
||||||
|
containerized)
|
||||||
|
- Why systemd: Survives Docker daemon restarts, no Docker socket mount
|
||||||
|
needed for host metrics, resource-limited via unit file
|
||||||
|
|
||||||
|
### Metrics Collected
|
||||||
|
|
||||||
|
- Docker memory per container (replaces existing bash cron)
|
||||||
|
- Host total memory usage
|
||||||
|
- Host swap usage
|
||||||
|
- Host CPU usage
|
||||||
|
- Apache active worker count
|
||||||
|
|
||||||
|
### Collection Architecture
|
||||||
|
|
||||||
|
- Two goroutines in a single binary:
|
||||||
|
- **Collector goroutine:** Gathers metrics every 30 seconds, appends JSON
|
||||||
|
line to `/var/log/pbs-monitor/metrics.jsonl`. Cannot be slowed by MySQL.
|
||||||
|
- **Sync goroutine:** Reads new JSONL entries every few minutes, bulk
|
||||||
|
INSERTs to MySQL. If MySQL is unavailable, retries next cycle. File still
|
||||||
|
has the data.
|
||||||
|
- Log rotation via standard logrotate config
|
||||||
|
|
||||||
|
### HTTP API Endpoints
|
||||||
|
|
||||||
|
- `GET /metrics/current` — latest metrics snapshot
|
||||||
|
- `GET /metrics/history?range=1h` — recent history from MySQL
|
||||||
|
- `GET /health` — service health check
|
||||||
|
- Binds to localhost only; n8n reaches via Docker host bridge IP
|
||||||
|
|
||||||
|
### Data Retention
|
||||||
|
|
||||||
|
- Raw 30-second samples: 30 days in `pbs_system_metrics`
|
||||||
|
- Hourly rollup: kept indefinitely in `pbs_system_metrics_hourly`
|
||||||
|
- n8n handles both the hourly aggregation job and the 30-day cleanup job
|
||||||
|
|
||||||
|
### Systemd Unit File
|
||||||
|
|
||||||
|
```ini
|
||||||
|
[Unit]
|
||||||
|
Description=PBS Monitor - system metrics collector
|
||||||
|
After=network.target docker.service
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
Type=simple
|
||||||
|
ExecStart=/usr/local/bin/pbs-monitor
|
||||||
|
Restart=on-failure
|
||||||
|
RestartSec=5s
|
||||||
|
User=pbs-monitor
|
||||||
|
Group=pbs-monitor
|
||||||
|
|
||||||
|
# Resource limits - blast radius containment
|
||||||
|
MemoryMax=256M
|
||||||
|
TasksMax=100
|
||||||
|
CPUQuota=20%
|
||||||
|
|
||||||
|
# Security hardening
|
||||||
|
NoNewPrivileges=true
|
||||||
|
ProtectSystem=strict
|
||||||
|
ProtectHome=true
|
||||||
|
ReadWritePaths=/var/log/pbs-monitor
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=multi-user.target
|
||||||
|
```
|
||||||
|
|
||||||
|
### Ansible Deployment
|
||||||
|
|
||||||
|
- New Ansible role for systemd service deployment
|
||||||
|
- Tasks: copy binary, create config, create system user (in docker group),
|
||||||
|
install unit file, install logrotate config, enable and start service
|
||||||
|
- Follows existing staging-first deployment pattern
|
||||||
|
|
||||||
|
## Log Parser (Python)
|
||||||
|
|
||||||
|
### Purpose
|
||||||
|
|
||||||
|
Parse WordPress Docker logs for security events and request patterns. Feeds
|
||||||
|
both the dashboard and Layer 3 alert rules.
|
||||||
|
|
||||||
|
### Technology
|
||||||
|
|
||||||
|
- Language: Python (UV + venv, PyCharm Professional)
|
||||||
|
- Orchestration: n8n schedules execution every 15 minutes
|
||||||
|
- Log source: `docker logs wordpress --since` command (no container changes
|
||||||
|
required)
|
||||||
|
|
||||||
|
### Incremental Parsing
|
||||||
|
|
||||||
|
- Query `MAX(logged_at)` from `pbs_security_events` to determine where to
|
||||||
|
start
|
||||||
|
- Use `--since` flag on `docker logs` to avoid reprocessing
|
||||||
|
- Self-healing on restart — just picks up from last stored timestamp
|
||||||
|
|
||||||
|
### Fields Extracted
|
||||||
|
|
||||||
|
- `logged_at` — timestamp
|
||||||
|
- `ip_address` — source IP
|
||||||
|
- `method` — GET/POST
|
||||||
|
- `uri` — request path
|
||||||
|
- `status_code` — HTTP response code
|
||||||
|
- `response_size` — bytes sent
|
||||||
|
- `user_agent` — client user agent
|
||||||
|
- `event_type` — derived category (e.g., `login_attempt`, `404`, `normal`)
|
||||||
|
- Note: `response_time_ms` deferred — requires custom Apache LogFormat, not
|
||||||
|
needed for v1
|
||||||
|
|
||||||
|
### MySQL Tables
|
||||||
|
|
||||||
|
#### pbs_security_events (raw log entries)
|
||||||
|
|
||||||
|
```sql
|
||||||
|
CREATE TABLE pbs_security_events (
|
||||||
|
id BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
|
||||||
|
logged_at DATETIME NOT NULL,
|
||||||
|
ip_address VARCHAR(45) NOT NULL,
|
||||||
|
method VARCHAR(10),
|
||||||
|
uri TEXT,
|
||||||
|
status_code SMALLINT,
|
||||||
|
response_size INT,
|
||||||
|
user_agent TEXT,
|
||||||
|
event_type VARCHAR(50),
|
||||||
|
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
INDEX idx_logged_at (logged_at),
|
||||||
|
INDEX idx_ip (ip_address),
|
||||||
|
INDEX idx_uri (uri(100))
|
||||||
|
);
|
||||||
|
```
|
||||||
|
|
||||||
|
#### pbs_security_summary (hourly aggregates)
|
||||||
|
|
||||||
|
```sql
|
||||||
|
CREATE TABLE pbs_security_summary (
|
||||||
|
id BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
|
||||||
|
hour_bucket DATETIME NOT NULL,
|
||||||
|
total_requests INT,
|
||||||
|
unique_ips INT,
|
||||||
|
login_attempts INT,
|
||||||
|
login_page_loads INT,
|
||||||
|
status_2xx_count INT,
|
||||||
|
status_3xx_count INT,
|
||||||
|
status_4xx_count INT,
|
||||||
|
status_404_count INT,
|
||||||
|
status_5xx_count INT,
|
||||||
|
top_ip VARCHAR(45),
|
||||||
|
top_ip_count INT,
|
||||||
|
avg_requests_per_minute DECIMAL(10,2),
|
||||||
|
peak_requests_per_minute INT,
|
||||||
|
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
UNIQUE INDEX idx_hour (hour_bucket)
|
||||||
|
);
|
||||||
|
```
|
||||||
|
|
||||||
|
- Hourly rollup performed by n8n workflow (separate from the parser)
|
||||||
|
- Raw events retention: grow unbounded initially, add cleanup policy when
|
||||||
|
volume is understood
|
||||||
|
- Summary table retention: indefinite
|
||||||
|
|
||||||
|
## Layer 2: Dashboard (Streamlit)
|
||||||
|
|
||||||
|
### Purpose
|
||||||
|
|
||||||
|
Visualize system health and security data. Read-only, human-facing.
|
||||||
|
|
||||||
|
### Technology
|
||||||
|
|
||||||
|
- Streamlit, Python, reads from MySQL
|
||||||
|
- Charts via Plotly or Altair (time series)
|
||||||
|
|
||||||
|
### Pages
|
||||||
|
|
||||||
|
#### Page 1: System Metrics
|
||||||
|
|
||||||
|
- Time series graphs: Docker memory per container, host memory, swap, CPU,
|
||||||
|
Apache workers
|
||||||
|
- Horizontal threshold lines showing Layer 3 warning and healer trigger
|
||||||
|
levels
|
||||||
|
- Time range selector: 24h (default), 7 days, 30 days
|
||||||
|
|
||||||
|
#### Page 2: Security / Log Events
|
||||||
|
|
||||||
|
- wp-login.php attempts over time
|
||||||
|
- Top attacking IPs (table)
|
||||||
|
- Status code distribution
|
||||||
|
- Request volume trends (avg and peak requests/min)
|
||||||
|
|
||||||
|
#### Page 3: Healer Activity
|
||||||
|
|
||||||
|
- Log of healer invocations: trigger, decision, action, outcome
|
||||||
|
- Timeline view of actions taken
|
||||||
|
- Deferred actions and their resolution (executed, auto-cancelled,
|
||||||
|
overridden)
|
||||||
|
|
||||||
|
## Layer 3: Monitor (n8n)
|
||||||
|
|
||||||
|
### Purpose
|
||||||
|
|
||||||
|
Early warning system. Detects degradation trends before they become
|
||||||
|
outages. Evaluates rules, fires alerts or invokes the healer.
|
||||||
|
|
||||||
|
### Mechanism
|
||||||
|
|
||||||
|
- n8n workflow, scheduled every 5 minutes
|
||||||
|
- Queries MySQL for recent metrics
|
||||||
|
- Evaluates threshold rules
|
||||||
|
- Works alongside Uptime Kuma (not replacing it for now)
|
||||||
|
|
||||||
|
### Starter Rules
|
||||||
|
|
||||||
|
| Rule | Threshold | Action |
|
||||||
|
|---|---|---|
|
||||||
|
| Host RAM usage | >85% sustained 10 min | Google Chat warning |
|
||||||
|
| Host RAM usage | >95% sustained 5 min | Invoke healer |
|
||||||
|
| Swap usage | >50% of total swap | Google Chat warning |
|
||||||
|
| Container memory | >90% of cap sustained 5 min | Google Chat warning |
|
||||||
|
| Container memory | >98% of cap sustained 3 min | Invoke healer |
|
||||||
|
| Host CPU | >80% sustained 15 min | Google Chat warning (info only) |
|
||||||
|
| Apache workers | >80% of max sustained 10 min | Google Chat warning |
|
||||||
|
| wp-login.php POSTs | >30 in 5-min window | Google Chat warning |
|
||||||
|
|
||||||
|
"Sustained" = all samples in the window exceed the threshold. Prevents
|
||||||
|
single-spike false positives.
|
||||||
|
|
||||||
|
### Alert Tiers
|
||||||
|
|
||||||
|
- **Warning** — Google Chat notification, no action taken
|
||||||
|
- **Invoke Healer** — hands off to Layer 4 with trigger context
|
||||||
|
|
||||||
|
## Layer 4: Healer (n8n + Claude API)
|
||||||
|
|
||||||
|
### Purpose
|
||||||
|
|
||||||
|
Take corrective action when Layer 3 invokes it. Reason about the problem,
|
||||||
|
execute a safe response, report results.
|
||||||
|
|
||||||
|
### Mechanism
|
||||||
|
|
||||||
|
- n8n workflow triggered by Layer 3 or Uptime Kuma webhooks
|
||||||
|
- Portainer REST API for container operations
|
||||||
|
- Claude API for diagnostic reasoning
|
||||||
|
|
||||||
|
### Available Actions
|
||||||
|
|
||||||
|
| Action | Timing | Safety Rails |
|
||||||
|
|---|---|---|
|
||||||
|
| Container restart (non-protected) | Immediate | Max 1/hour, 3/24h per
|
||||||
|
container |
|
||||||
|
| WordPress restart | Deferred to quiet hours | Same caps + admin override
|
||||||
|
window |
|
||||||
|
| MySQL restart | Deferred to quiet hours | Same caps + admin override
|
||||||
|
window |
|
||||||
|
| Server reboot | Deferred to quiet hours | Max 1/24h + admin override
|
||||||
|
window |
|
||||||
|
| Alert only | Immediate | Always an option |
|
||||||
|
| Escalate | Immediate | When limits exceeded |
|
||||||
|
|
||||||
|
**Protected containers** (deferred to quiet hours only): WordPress, MySQL
|
||||||
|
|
||||||
|
### Quiet Hours
|
||||||
|
|
||||||
|
- Window: 0200–0500 ET
|
||||||
|
- Deferred actions are standing orders, not scheduled events
|
||||||
|
- During quiet hours, n8n checks: is there a pending action? Is it still
|
||||||
|
pending (not overridden)? Do the original conditions still exist?
|
||||||
|
- If conditions resolved → auto-cancel with Google Chat notification
|
||||||
|
- If overridden → mark as overridden
|
||||||
|
|
||||||
|
### Deferred Action Flow
|
||||||
|
|
||||||
|
1. Healer decides a protected container or server needs restart
|
||||||
|
2. Sets flag in `pbs_healer_actions`: `deferred_action`, `deferred_status =
|
||||||
|
'pending'`
|
||||||
|
3. Google Chat notification with action ID and reasoning
|
||||||
|
4. During quiet hours, n8n evaluates:
|
||||||
|
- Pending action exists? → Check if overridden → Check if conditions
|
||||||
|
persist → Execute or auto-cancel
|
||||||
|
5. Status transitions: `pending` → `executed` | `auto_cancelled` |
|
||||||
|
`overridden`
|
||||||
|
|
||||||
|
### Override Mechanism
|
||||||
|
|
||||||
|
#### V1
|
||||||
|
|
||||||
|
- **pbs-hub web page:** Shows pending healer actions, click to
|
||||||
|
cancel/approve
|
||||||
|
- **Google Chat webhook:** Text command with action ID (e.g., "override
|
||||||
|
1234", "cancel 1234")
|
||||||
|
- Both hit the same pbs-hub API endpoint
|
||||||
|
- pbs-hub API also available for future integrations
|
||||||
|
|
||||||
|
#### V2 (future)
|
||||||
|
|
||||||
|
- Google Chat interactive cards with approve/cancel/defer buttons
|
||||||
|
|
||||||
|
### Iteration Limits (Anti-Loop Protection)
|
||||||
|
|
||||||
|
- Per-container cooldown: 1 hour between automatic restarts
|
||||||
|
- Per-container daily cap: 3 automatic restarts per 24 hours
|
||||||
|
- Global healer cap: 10 healer-initiated actions per 24 hours
|
||||||
|
- Cooldown on escalation: 6 hours of no auto-action for that container
|
||||||
|
after escalation
|
||||||
|
- If any limit hit → escalate instead of acting
|
||||||
|
|
||||||
|
### Healer Decision Flow
|
||||||
|
|
||||||
|
1. Receive trigger from Layer 3 or Uptime Kuma
|
||||||
|
2. Check iteration limits → if exceeded, escalate and stop
|
||||||
|
3. Gather context:
|
||||||
|
- Current metrics from pbs-monitor HTTP API
|
||||||
|
- Recent log events from pbs_security_events
|
||||||
|
- Container inspect + recent logs from Portainer API
|
||||||
|
4. Send structured context to Claude API
|
||||||
|
5. Claude returns: recommended action, reasoning, confidence
|
||||||
|
6. If action is allowed AND confidence is high → execute (or defer if
|
||||||
|
protected)
|
||||||
|
7. Else → escalate with Claude's reasoning
|
||||||
|
8. Log decision + action + outcome to `pbs_healer_actions`
|
||||||
|
9. Google Chat notification with: what happened, what healer did (or
|
||||||
|
didn't), Claude's reasoning
|
||||||
|
|
||||||
|
### Claude API Prompt Shape
|
||||||
|
|
||||||
|
- Structured input: trigger reason, current metrics snapshot, 30-min
|
||||||
|
history, recent logs, container inspect
|
||||||
|
- Structured output: diagnosis, recommended action (enum:
|
||||||
|
`restart_container`, `defer_restart`, `reboot_server`, `no_action`,
|
||||||
|
`escalate`), reasoning, confidence level
|
||||||
|
|
||||||
|
### pbs_healer_actions Table
|
||||||
|
|
||||||
|
```sql
|
||||||
|
CREATE TABLE pbs_healer_actions (
|
||||||
|
id BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
|
||||||
|
triggered_at DATETIME NOT NULL,
|
||||||
|
trigger_source VARCHAR(50),
|
||||||
|
trigger_rule VARCHAR(100),
|
||||||
|
container_name VARCHAR(100),
|
||||||
|
metrics_snapshot JSON,
|
||||||
|
claude_reasoning TEXT,
|
||||||
|
action_recommended VARCHAR(50),
|
||||||
|
action_taken VARCHAR(50),
|
||||||
|
action_result TEXT,
|
||||||
|
deferred_action VARCHAR(50),
|
||||||
|
deferred_status VARCHAR(20) DEFAULT NULL,
|
||||||
|
iteration_limit_hit BOOLEAN DEFAULT FALSE,
|
||||||
|
escalated BOOLEAN DEFAULT FALSE,
|
||||||
|
created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
INDEX idx_triggered (triggered_at),
|
||||||
|
INDEX idx_container (container_name),
|
||||||
|
INDEX idx_deferred (deferred_status)
|
||||||
|
);
|
||||||
|
```
|
||||||
|
|
||||||
|
### Starting Conservatism
|
||||||
|
|
||||||
|
- First period (target ~30 days): **dry-run mode** — healer makes
|
||||||
|
decisions, logs them, sends Google Chat with "I would have done X" but does
|
||||||
|
not execute
|
||||||
|
- After validation: flip to live mode, container restarts first
|
||||||
|
- Server reboot: enabled after container restart trust is established
|
||||||
|
- System reboot scheduling added only after override mechanism is validated
|
||||||
|
|
||||||
|
## Existing Infrastructure Context
|
||||||
|
|
||||||
|
- Docker/Traefik/Linode production stack
|
||||||
|
- Portainer running with REST API available
|
||||||
|
- Uptime Kuma operational for up/down checks and Google Chat alerts
|
||||||
|
- `pbs_automation` MySQL database exists with separate permissions
|
||||||
|
- n8n operational, Claude API integration is a known pattern
|
||||||
|
- Existing bash cron (60s, Docker memory only, flat file) — replaced by
|
||||||
|
Layer 1
|
||||||
|
- Crowdsec security hardening is a separate project (not duplicated here)
|
||||||
|
|
||||||
|
## Implementation Order
|
||||||
|
|
||||||
|
- [ ] Phase 1: Layer 1 — Go collector (systemd, JSONL, MySQL sync, HTTP API)
|
||||||
|
- [ ] Phase 2: Log Parser — Python WordPress log parser + MySQL schema
|
||||||
|
- [ ] Phase 3: Layer 2 — Streamlit dashboard (system metrics + security +
|
||||||
|
thresholds)
|
||||||
|
- [ ] Phase 4: Layer 3 — n8n monitor workflow (threshold rules, Google Chat
|
||||||
|
alerts)
|
||||||
|
- [ ] Phase 5: Layer 4 — n8n healer workflow (dry-run mode, Claude API
|
||||||
|
integration)
|
||||||
|
- [ ] Phase 6: Layer 4 live — enable live actions, override page in pbs-hub
|
||||||
|
- [ ] Phase 7: Deferred actions — quiet hours logic, WordPress/MySQL/server
|
||||||
|
restart scheduling
|
||||||
|
|
||||||
|
## Open Questions
|
||||||
|
|
||||||
|
- [ ] Go project structure — standard layout or flat? Decide during Phase 1
|
||||||
|
- [ ] Docker multi-stage build for Go binary — needed for CI, decide during
|
||||||
|
Phase 1
|
||||||
|
- [ ] MySQL schema for `pbs_system_metrics` and `pbs_system_metrics_hourly`
|
||||||
|
— design during Phase 1
|
||||||
|
- [ ] Apache max_workers value — needed to calculate percentage for Layer 3
|
||||||
|
rules
|
||||||
|
- [ ] Portainer API auth — how is it currently secured? Needed for Layer 4
|
||||||
|
- [ ] Claude API prompt engineering — detailed prompt design during Phase 5
|
||||||
|
- [ ] pbs-hub override page routing — how does it fit into existing pbs-hub
|
||||||
|
Flask app?
|
||||||
|
- [ ] Google Chat webhook for override commands — new webhook or extend
|
||||||
|
existing?
|
||||||
|
- [ ] Streamlit deployment — containerized or host-level? Decide during
|
||||||
|
Phase 3
|
||||||
|
|
||||||
|
...sent from Jenny & Travis
|
||||||
Loading…
Reference in New Issue
Block a user