pbs-projects/PBS/Tech/Projects/pbs-infrastructure-intelligence.md

---
project: pbs-infrastructure-intelligence
type: project-plan
status: active
path: PBS/Tech/Projects
tags:
  - pbs
  - monitoring
  - docker
  - go
  - python
  - mysql
  - n8n
  - streamlit
  - security
created: 2026-04-16
updated: 2026-04-16
---

# PBS Infrastructure Intelligence (PBSII)

## Goal

Build a self-healing, intelligent monitoring system for the PBS
infrastructure. Four interconnected layers: a metrics collector, a
visualization dashboard, a threshold monitor, and an AI-powered healer.

## Problem Being Solved

- DoS attacks cause server thrashing with no early warning
- Container memory leaks go undetected until crashes occur
- Existing monitoring (Uptime Kuma) only detects hard-down states, not
degradation trends
- Manual log grepping is unsustainable as the site grows
- No visibility into what traffic reaches the server vs what Cloudflare
blocks
- Apache worker exhaustion causes memory crashes with no forensic data

## Architecture Overview

```
Layer 1: Collector (Go, systemd)
  → JSONL file (30s interval)
  → MySQL sync (every few minutes)
  → HTTP API (/metrics/current, /metrics/history, /health)

Log Parser (Python, n8n-scheduled)
  → docker logs --since
  → MySQL pbs_security_events + pbs_security_summary

Layer 2: Dashboard (Streamlit)
  → Reads MySQL
  → System metrics + security events + healer activity

Layer 3: Monitor (n8n, every 5 min)
  → Evaluates threshold rules against MySQL data
  → Fires warnings (Google Chat) or invokes healer

Layer 4: Healer (n8n + Claude API)
  → Gathers diagnostic context from all layers
  → Claude API reasons about the problem
  → Executes allowed actions or escalates
  → Logs all decisions to pbs_healer_actions
```

## Layer 1: Collector (Go Microservice)

### Purpose

Collect system and container metrics. Write to disk first, sync to MySQL
second. Serve current state via HTTP. No opinions, just data.

### Technology

- Language: Go (standard library first)
- Deployment: Single static binary, systemd service on host (not
containerized)
- Why systemd: Survives Docker daemon restarts, no Docker socket mount
needed for host metrics, resource-limited via unit file

### Metrics Collected

- Docker memory per container (replaces existing bash cron)
- Host total memory usage
- Host swap usage
- Host CPU usage
- Apache active worker count

### Collection Architecture

- Two goroutines in a single binary:
  - **Collector goroutine:** Gathers metrics every 30 seconds, appends JSON
line to `/var/log/pbs-monitor/metrics.jsonl`. Cannot be slowed by MySQL.
  - **Sync goroutine:** Reads new JSONL entries every few minutes, bulk
INSERTs to MySQL. If MySQL is unavailable, retries next cycle. File still
has the data.
- Log rotation via standard logrotate config

### HTTP API Endpoints

- `GET /metrics/current` — latest metrics snapshot
- `GET /metrics/history?range=1h` — recent history from MySQL
- `GET /health` — service health check
- Binds to localhost only; n8n reaches via Docker host bridge IP

### Data Retention

- Raw 30-second samples: 30 days in `pbs_system_metrics`
- Hourly rollup: kept indefinitely in `pbs_system_metrics_hourly`
- n8n handles both the hourly aggregation job and the 30-day cleanup job

### Systemd Unit File

```ini
[Unit]
Description=PBS Monitor - system metrics collector
After=network.target docker.service

[Service]
Type=simple
ExecStart=/usr/local/bin/pbs-monitor
Restart=on-failure
RestartSec=5s
User=pbs-monitor
Group=pbs-monitor

# Resource limits - blast radius containment
MemoryMax=256M
TasksMax=100
CPUQuota=20%

# Security hardening
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/log/pbs-monitor

[Install]
WantedBy=multi-user.target
```

### Ansible Deployment

- New Ansible role for systemd service deployment
- Tasks: copy binary, create config, create system user (in docker group),
install unit file, install logrotate config, enable and start service
- Follows existing staging-first deployment pattern

## Log Parser (Python)

### Purpose

Parse WordPress Docker logs for security events and request patterns. Feeds
both the dashboard and Layer 3 alert rules.

### Technology

- Language: Python (UV + venv, PyCharm Professional)
- Orchestration: n8n schedules execution every 15 minutes
- Log source: `docker logs wordpress --since` command (no container changes
required)

### Incremental Parsing

- Query `MAX(logged_at)` from `pbs_security_events` to determine where to
start
- Use `--since` flag on `docker logs` to avoid reprocessing
- Self-healing on restart — just picks up from last stored timestamp

### Fields Extracted

- `logged_at` — timestamp
- `ip_address` — source IP
- `method` — GET/POST
- `uri` — request path
- `status_code` — HTTP response code
- `response_size` — bytes sent
- `user_agent` — client user agent
- `event_type` — derived category (e.g., `login_attempt`, `404`, `normal`)
- Note: `response_time_ms` deferred — requires custom Apache LogFormat, not
needed for v1

### MySQL Tables

#### pbs_security_events (raw log entries)

```sql
CREATE TABLE pbs_security_events (
    id BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
    logged_at DATETIME NOT NULL,
    ip_address VARCHAR(45) NOT NULL,
    method VARCHAR(10),
    uri TEXT,
    status_code SMALLINT,
    response_size INT,
    user_agent TEXT,
    event_type VARCHAR(50),
    created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
    INDEX idx_logged_at (logged_at),
    INDEX idx_ip (ip_address),
    INDEX idx_uri (uri(100))
);
```

#### pbs_security_summary (hourly aggregates)

```sql
CREATE TABLE pbs_security_summary (
    id BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
    hour_bucket DATETIME NOT NULL,
    total_requests INT,
    unique_ips INT,
    login_attempts INT,
    login_page_loads INT,
    status_2xx_count INT,
    status_3xx_count INT,
    status_4xx_count INT,
    status_404_count INT,
    status_5xx_count INT,
    top_ip VARCHAR(45),
    top_ip_count INT,
    avg_requests_per_minute DECIMAL(10,2),
    peak_requests_per_minute INT,
    created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
    UNIQUE INDEX idx_hour (hour_bucket)
);
```

- Hourly rollup performed by n8n workflow (separate from the parser)
- Raw events retention: grow unbounded initially, add cleanup policy when
volume is understood
- Summary table retention: indefinite

## Layer 2: Dashboard (Streamlit)

### Purpose

Visualize system health and security data. Read-only, human-facing.

### Technology

- Streamlit, Python, reads from MySQL
- Charts via Plotly or Altair (time series)

### Pages

#### Page 1: System Metrics

- Time series graphs: Docker memory per container, host memory, swap, CPU,
Apache workers
- Horizontal threshold lines showing Layer 3 warning and healer trigger
levels
- Time range selector: 24h (default), 7 days, 30 days

#### Page 2: Security / Log Events

- wp-login.php attempts over time
- Top attacking IPs (table)
- Status code distribution
- Request volume trends (avg and peak requests/min)

#### Page 3: Healer Activity

- Log of healer invocations: trigger, decision, action, outcome
- Timeline view of actions taken
- Deferred actions and their resolution (executed, auto-cancelled,
overridden)

## Layer 3: Monitor (n8n)

### Purpose

Early warning system. Detects degradation trends before they become
outages. Evaluates rules, fires alerts or invokes the healer.

### Mechanism

- n8n workflow, scheduled every 5 minutes
- Queries MySQL for recent metrics
- Evaluates threshold rules
- Works alongside Uptime Kuma (not replacing it for now)

### Starter Rules

| Rule | Threshold | Action |
|---|---|---|
| Host RAM usage | >85% sustained 10 min | Google Chat warning |
| Host RAM usage | >95% sustained 5 min | Invoke healer |
| Swap usage | >50% of total swap | Google Chat warning |
| Container memory | >90% of cap sustained 5 min | Google Chat warning |
| Container memory | >98% of cap sustained 3 min | Invoke healer |
| Host CPU | >80% sustained 15 min | Google Chat warning (info only) |
| Apache workers | >80% of max sustained 10 min | Google Chat warning |
| wp-login.php POSTs | >30 in 5-min window | Google Chat warning |

"Sustained" = all samples in the window exceed the threshold. Prevents
single-spike false positives.

### Alert Tiers

- **Warning** — Google Chat notification, no action taken
- **Invoke Healer** — hands off to Layer 4 with trigger context

## Layer 4: Healer (n8n + Claude API)

### Purpose

Take corrective action when Layer 3 invokes it. Reason about the problem,
execute a safe response, report results.

### Mechanism

- n8n workflow triggered by Layer 3 or Uptime Kuma webhooks
- Portainer REST API for container operations
- Claude API for diagnostic reasoning

### Available Actions

| Action | Timing | Safety Rails |
|---|---|---|
| Container restart (non-protected) | Immediate | Max 1/hour, 3/24h per
container |
| WordPress restart | Deferred to quiet hours | Same caps + admin override
window |
| MySQL restart | Deferred to quiet hours | Same caps + admin override
window |
| Server reboot | Deferred to quiet hours | Max 1/24h + admin override
window |
| Alert only | Immediate | Always an option |
| Escalate | Immediate | When limits exceeded |

**Protected containers** (deferred to quiet hours only): WordPress, MySQL

### Quiet Hours

- Window: 0200–0500 ET
- Deferred actions are standing orders, not scheduled events
- During quiet hours, n8n checks: is there a pending action? Is it still
pending (not overridden)? Do the original conditions still exist?
- If conditions resolved → auto-cancel with Google Chat notification
- If overridden → mark as overridden

### Deferred Action Flow

1. Healer decides a protected container or server needs restart
2. Sets flag in `pbs_healer_actions`: `deferred_action`, `deferred_status =
'pending'`
3. Google Chat notification with action ID and reasoning
4. During quiet hours, n8n evaluates:
   - Pending action exists? → Check if overridden → Check if conditions
persist → Execute or auto-cancel
5. Status transitions: `pending` → `executed` | `auto_cancelled` |
`overridden`

### Override Mechanism

#### V1

- **pbs-hub web page:** Shows pending healer actions, click to
cancel/approve
- **Google Chat webhook:** Text command with action ID (e.g., "override
1234", "cancel 1234")
- Both hit the same pbs-hub API endpoint
- pbs-hub API also available for future integrations

#### V2 (future)

- Google Chat interactive cards with approve/cancel/defer buttons

### Iteration Limits (Anti-Loop Protection)

- Per-container cooldown: 1 hour between automatic restarts
- Per-container daily cap: 3 automatic restarts per 24 hours
- Global healer cap: 10 healer-initiated actions per 24 hours
- Cooldown on escalation: 6 hours of no auto-action for that container
after escalation
- If any limit hit → escalate instead of acting

### Healer Decision Flow

1. Receive trigger from Layer 3 or Uptime Kuma
2. Check iteration limits → if exceeded, escalate and stop
3. Gather context:
   - Current metrics from pbs-monitor HTTP API
   - Recent log events from pbs_security_events
   - Container inspect + recent logs from Portainer API
4. Send structured context to Claude API
5. Claude returns: recommended action, reasoning, confidence
6. If action is allowed AND confidence is high → execute (or defer if
protected)
7. Else → escalate with Claude's reasoning
8. Log decision + action + outcome to `pbs_healer_actions`
9. Google Chat notification with: what happened, what healer did (or
didn't), Claude's reasoning

### Claude API Prompt Shape

- Structured input: trigger reason, current metrics snapshot, 30-min
history, recent logs, container inspect
- Structured output: diagnosis, recommended action (enum:
`restart_container`, `defer_restart`, `reboot_server`, `no_action`,
`escalate`), reasoning, confidence level

### pbs_healer_actions Table

```sql
CREATE TABLE pbs_healer_actions (
    id BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
    triggered_at DATETIME NOT NULL,
    trigger_source VARCHAR(50),
    trigger_rule VARCHAR(100),
    container_name VARCHAR(100),
    metrics_snapshot JSON,
    claude_reasoning TEXT,
    action_recommended VARCHAR(50),
    action_taken VARCHAR(50),
    action_result TEXT,
    deferred_action VARCHAR(50),
    deferred_status VARCHAR(20) DEFAULT NULL,
    iteration_limit_hit BOOLEAN DEFAULT FALSE,
    escalated BOOLEAN DEFAULT FALSE,
    created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
    INDEX idx_triggered (triggered_at),
    INDEX idx_container (container_name),
    INDEX idx_deferred (deferred_status)
);
```

### Starting Conservatism

- First period (target ~30 days): **dry-run mode** — healer makes
decisions, logs them, sends Google Chat with "I would have done X" but does
not execute
- After validation: flip to live mode, container restarts first
- Server reboot: enabled after container restart trust is established
- System reboot scheduling added only after override mechanism is validated

## Existing Infrastructure Context

- Docker/Traefik/Linode production stack
- Portainer running with REST API available
- Uptime Kuma operational for up/down checks and Google Chat alerts
- `pbs_automation` MySQL database exists with separate permissions
- n8n operational, Claude API integration is a known pattern
- Existing bash cron (60s, Docker memory only, flat file) — replaced by
Layer 1
- Crowdsec security hardening is a separate project (not duplicated here)

## Implementation Order

- [ ] Phase 1: Layer 1 — Go collector (systemd, JSONL, MySQL sync, HTTP API)
- [ ] Phase 2: Log Parser — Python WordPress log parser + MySQL schema
- [ ] Phase 3: Layer 2 — Streamlit dashboard (system metrics + security +
thresholds)
- [ ] Phase 4: Layer 3 — n8n monitor workflow (threshold rules, Google Chat
alerts)
- [ ] Phase 5: Layer 4 — n8n healer workflow (dry-run mode, Claude API
integration)
- [ ] Phase 6: Layer 4 live — enable live actions, override page in pbs-hub
- [ ] Phase 7: Deferred actions — quiet hours logic, WordPress/MySQL/server
restart scheduling

## Open Questions

- [ ] Go project structure — standard layout or flat? Decide during Phase 1
- [ ] Docker multi-stage build for Go binary — needed for CI, decide during
Phase 1
- [ ] MySQL schema for `pbs_system_metrics` and `pbs_system_metrics_hourly`
— design during Phase 1
- [ ] Apache max_workers value — needed to calculate percentage for Layer 3
rules
- [ ] Portainer API auth — how is it currently secured? Needed for Layer 4
- [ ] Claude API prompt engineering — detailed prompt design during Phase 5
- [ ] pbs-hub override page routing — how does it fit into existing pbs-hub
Flask app?
- [ ] Google Chat webhook for override commands — new webhook or extend
existing?
- [ ] Streamlit deployment — containerized or host-level? Decide during
Phase 3

...sent from Jenny & Travis