herbygitea/pbs-projects
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
1d352fe606
pbs-projects/PBS/Tech/Projects/pbs-infrastructure-intelligence.md

15 KiB
Raw Blame History

project type status path tags created updated
pbs-infrastructure-intelligence project-plan active PBS/Tech/Projects
pbs
monitoring
docker
go
python
mysql
n8n
streamlit
security
2026-04-16 2026-04-16

PBS Infrastructure Intelligence (PBSII)

Goal

Build a self-healing, intelligent monitoring system for the PBS infrastructure. Four interconnected layers: a metrics collector, a visualization dashboard, a threshold monitor, and an AI-powered healer.

Problem Being Solved

  • DoS attacks cause server thrashing with no early warning
  • Container memory leaks go undetected until crashes occur
  • Existing monitoring (Uptime Kuma) only detects hard-down states, not degradation trends
  • Manual log grepping is unsustainable as the site grows
  • No visibility into what traffic reaches the server vs what Cloudflare blocks
  • Apache worker exhaustion causes memory crashes with no forensic data

Architecture Overview

Layer 1: Collector (Go, systemd)
  → JSONL file (30s interval)
  → MySQL sync (every few minutes)
  → HTTP API (/metrics/current, /metrics/history, /health)

Log Parser (Python, n8n-scheduled)
  → docker logs --since
  → MySQL pbs_security_events + pbs_security_summary

Layer 2: Dashboard (Streamlit)
  → Reads MySQL
  → System metrics + security events + healer activity

Layer 3: Monitor (n8n, every 5 min)
  → Evaluates threshold rules against MySQL data
  → Fires warnings (Google Chat) or invokes healer

Layer 4: Healer (n8n + Claude API)
  → Gathers diagnostic context from all layers
  → Claude API reasons about the problem
  → Executes allowed actions or escalates
  → Logs all decisions to pbs_healer_actions

Layer 1: Collector (Go Microservice)

Purpose

Collect system and container metrics. Write to disk first, sync to MySQL second. Serve current state via HTTP. No opinions, just data.

Technology

  • Language: Go (standard library first)
  • Deployment: Single static binary, systemd service on host (not containerized)
  • Why systemd: Survives Docker daemon restarts, no Docker socket mount needed for host metrics, resource-limited via unit file

Metrics Collected

  • Docker memory per container (replaces existing bash cron)
  • Host total memory usage
  • Host swap usage
  • Host CPU usage
  • Apache active worker count

Collection Architecture

  • Two goroutines in a single binary:
    • Collector goroutine: Gathers metrics every 30 seconds, appends JSON line to /var/log/pbs-monitor/metrics.jsonl. Cannot be slowed by MySQL.
    • Sync goroutine: Reads new JSONL entries every few minutes, bulk INSERTs to MySQL. If MySQL is unavailable, retries next cycle. File still has the data.
  • Log rotation via standard logrotate config

HTTP API Endpoints

  • GET /metrics/current — latest metrics snapshot
  • GET /metrics/history?range=1h — recent history from MySQL
  • GET /health — service health check
  • Binds to localhost only; n8n reaches via Docker host bridge IP

Data Retention

  • Raw 30-second samples: 30 days in pbs_system_metrics
  • Hourly rollup: kept indefinitely in pbs_system_metrics_hourly
  • n8n handles both the hourly aggregation job and the 30-day cleanup job

Systemd Unit File

[Unit]
Description=PBS Monitor - system metrics collector
After=network.target docker.service

[Service]
Type=simple
ExecStart=/usr/local/bin/pbs-monitor
Restart=on-failure
RestartSec=5s
User=pbs-monitor
Group=pbs-monitor

# Resource limits - blast radius containment
MemoryMax=256M
TasksMax=100
CPUQuota=20%

# Security hardening
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/log/pbs-monitor

[Install]
WantedBy=multi-user.target

Ansible Deployment

  • New Ansible role for systemd service deployment
  • Tasks: copy binary, create config, create system user (in docker group), install unit file, install logrotate config, enable and start service
  • Follows existing staging-first deployment pattern

Log Parser (Python)

Purpose

Parse WordPress Docker logs for security events and request patterns. Feeds both the dashboard and Layer 3 alert rules.

Technology

  • Language: Python (UV + venv, PyCharm Professional)
  • Orchestration: n8n schedules execution every 15 minutes
  • Log source: docker logs wordpress --since command (no container changes required)

Incremental Parsing

  • Query MAX(logged_at) from pbs_security_events to determine where to start
  • Use --since flag on docker logs to avoid reprocessing
  • Self-healing on restart — just picks up from last stored timestamp

Fields Extracted

  • logged_at — timestamp
  • ip_address — source IP
  • method — GET/POST
  • uri — request path
  • status_code — HTTP response code
  • response_size — bytes sent
  • user_agent — client user agent
  • event_type — derived category (e.g., login_attempt, 404, normal)
  • Note: response_time_ms deferred — requires custom Apache LogFormat, not needed for v1

MySQL Tables

pbs_security_events (raw log entries)

CREATE TABLE pbs_security_events (
    id BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
    logged_at DATETIME NOT NULL,
    ip_address VARCHAR(45) NOT NULL,
    method VARCHAR(10),
    uri TEXT,
    status_code SMALLINT,
    response_size INT,
    user_agent TEXT,
    event_type VARCHAR(50),
    created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
    INDEX idx_logged_at (logged_at),
    INDEX idx_ip (ip_address),
    INDEX idx_uri (uri(100))
);

pbs_security_summary (hourly aggregates)

CREATE TABLE pbs_security_summary (
    id BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
    hour_bucket DATETIME NOT NULL,
    total_requests INT,
    unique_ips INT,
    login_attempts INT,
    login_page_loads INT,
    status_2xx_count INT,
    status_3xx_count INT,
    status_4xx_count INT,
    status_404_count INT,
    status_5xx_count INT,
    top_ip VARCHAR(45),
    top_ip_count INT,
    avg_requests_per_minute DECIMAL(10,2),
    peak_requests_per_minute INT,
    created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
    UNIQUE INDEX idx_hour (hour_bucket)
);
  • Hourly rollup performed by n8n workflow (separate from the parser)
  • Raw events retention: grow unbounded initially, add cleanup policy when volume is understood
  • Summary table retention: indefinite

Layer 2: Dashboard (Streamlit)

Purpose

Visualize system health and security data. Read-only, human-facing.

Technology

  • Streamlit, Python, reads from MySQL
  • Charts via Plotly or Altair (time series)

Pages

Page 1: System Metrics

  • Time series graphs: Docker memory per container, host memory, swap, CPU, Apache workers
  • Horizontal threshold lines showing Layer 3 warning and healer trigger levels
  • Time range selector: 24h (default), 7 days, 30 days

Page 2: Security / Log Events

  • wp-login.php attempts over time
  • Top attacking IPs (table)
  • Status code distribution
  • Request volume trends (avg and peak requests/min)

Page 3: Healer Activity

  • Log of healer invocations: trigger, decision, action, outcome
  • Timeline view of actions taken
  • Deferred actions and their resolution (executed, auto-cancelled, overridden)

Layer 3: Monitor (n8n)

Purpose

Early warning system. Detects degradation trends before they become outages. Evaluates rules, fires alerts or invokes the healer.

Mechanism

  • n8n workflow, scheduled every 5 minutes
  • Queries MySQL for recent metrics
  • Evaluates threshold rules
  • Works alongside Uptime Kuma (not replacing it for now)

Starter Rules

Rule Threshold Action
Host RAM usage >85% sustained 10 min Google Chat warning
Host RAM usage >95% sustained 5 min Invoke healer
Swap usage >50% of total swap Google Chat warning
Container memory >90% of cap sustained 5 min Google Chat warning
Container memory >98% of cap sustained 3 min Invoke healer
Host CPU >80% sustained 15 min Google Chat warning (info only)
Apache workers >80% of max sustained 10 min Google Chat warning
wp-login.php POSTs >30 in 5-min window Google Chat warning

"Sustained" = all samples in the window exceed the threshold. Prevents single-spike false positives.

Alert Tiers

  • Warning — Google Chat notification, no action taken
  • Invoke Healer — hands off to Layer 4 with trigger context

Layer 4: Healer (n8n + Claude API)

Purpose

Take corrective action when Layer 3 invokes it. Reason about the problem, execute a safe response, report results.

Mechanism

  • n8n workflow triggered by Layer 3 or Uptime Kuma webhooks
  • Portainer REST API for container operations
  • Claude API for diagnostic reasoning

Available Actions

Action Timing Safety Rails
Container restart (non-protected) Immediate Max 1/hour, 3/24h per
container
WordPress restart Deferred to quiet hours Same caps + admin override
window
MySQL restart Deferred to quiet hours Same caps + admin override
window
Server reboot Deferred to quiet hours Max 1/24h + admin override
window
Alert only Immediate Always an option
Escalate Immediate When limits exceeded

Protected containers (deferred to quiet hours only): WordPress, MySQL

Quiet Hours

  • Window: 0200–0500 ET
  • Deferred actions are standing orders, not scheduled events
  • During quiet hours, n8n checks: is there a pending action? Is it still pending (not overridden)? Do the original conditions still exist?
  • If conditions resolved → auto-cancel with Google Chat notification
  • If overridden → mark as overridden

Deferred Action Flow

  1. Healer decides a protected container or server needs restart
  2. Sets flag in pbs_healer_actions: deferred_action, deferred_status = 'pending'
  3. Google Chat notification with action ID and reasoning
  4. During quiet hours, n8n evaluates:
    • Pending action exists? → Check if overridden → Check if conditions persist → Execute or auto-cancel
  5. Status transitions: pending → executed | auto_cancelled | overridden

Override Mechanism

V1

  • pbs-hub web page: Shows pending healer actions, click to cancel/approve
  • Google Chat webhook: Text command with action ID (e.g., "override 1234", "cancel 1234")
  • Both hit the same pbs-hub API endpoint
  • pbs-hub API also available for future integrations

V2 (future)

  • Google Chat interactive cards with approve/cancel/defer buttons

Iteration Limits (Anti-Loop Protection)

  • Per-container cooldown: 1 hour between automatic restarts
  • Per-container daily cap: 3 automatic restarts per 24 hours
  • Global healer cap: 10 healer-initiated actions per 24 hours
  • Cooldown on escalation: 6 hours of no auto-action for that container after escalation
  • If any limit hit → escalate instead of acting

Healer Decision Flow

  1. Receive trigger from Layer 3 or Uptime Kuma
  2. Check iteration limits → if exceeded, escalate and stop
  3. Gather context:
    • Current metrics from pbs-monitor HTTP API
    • Recent log events from pbs_security_events
    • Container inspect + recent logs from Portainer API
  4. Send structured context to Claude API
  5. Claude returns: recommended action, reasoning, confidence
  6. If action is allowed AND confidence is high → execute (or defer if protected)
  7. Else → escalate with Claude's reasoning
  8. Log decision + action + outcome to pbs_healer_actions
  9. Google Chat notification with: what happened, what healer did (or didn't), Claude's reasoning

Claude API Prompt Shape

  • Structured input: trigger reason, current metrics snapshot, 30-min history, recent logs, container inspect
  • Structured output: diagnosis, recommended action (enum: restart_container, defer_restart, reboot_server, no_action, escalate), reasoning, confidence level

pbs_healer_actions Table

CREATE TABLE pbs_healer_actions (
    id BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
    triggered_at DATETIME NOT NULL,
    trigger_source VARCHAR(50),
    trigger_rule VARCHAR(100),
    container_name VARCHAR(100),
    metrics_snapshot JSON,
    claude_reasoning TEXT,
    action_recommended VARCHAR(50),
    action_taken VARCHAR(50),
    action_result TEXT,
    deferred_action VARCHAR(50),
    deferred_status VARCHAR(20) DEFAULT NULL,
    iteration_limit_hit BOOLEAN DEFAULT FALSE,
    escalated BOOLEAN DEFAULT FALSE,
    created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
    INDEX idx_triggered (triggered_at),
    INDEX idx_container (container_name),
    INDEX idx_deferred (deferred_status)
);

Starting Conservatism

  • First period (target ~30 days): dry-run mode — healer makes decisions, logs them, sends Google Chat with "I would have done X" but does not execute
  • After validation: flip to live mode, container restarts first
  • Server reboot: enabled after container restart trust is established
  • System reboot scheduling added only after override mechanism is validated

Existing Infrastructure Context

  • Docker/Traefik/Linode production stack
  • Portainer running with REST API available
  • Uptime Kuma operational for up/down checks and Google Chat alerts
  • pbs_automation MySQL database exists with separate permissions
  • n8n operational, Claude API integration is a known pattern
  • Existing bash cron (60s, Docker memory only, flat file) — replaced by Layer 1
  • Crowdsec security hardening is a separate project (not duplicated here)

Implementation Order

  • Phase 1: Layer 1 — Go collector (systemd, JSONL, MySQL sync, HTTP API)
  • Phase 2: Log Parser — Python WordPress log parser + MySQL schema
  • Phase 3: Layer 2 — Streamlit dashboard (system metrics + security + thresholds)
  • Phase 4: Layer 3 — n8n monitor workflow (threshold rules, Google Chat alerts)
  • Phase 5: Layer 4 — n8n healer workflow (dry-run mode, Claude API integration)
  • Phase 6: Layer 4 live — enable live actions, override page in pbs-hub
  • Phase 7: Deferred actions — quiet hours logic, WordPress/MySQL/server restart scheduling

Open Questions

  • Go project structure — standard layout or flat? Decide during Phase 1
  • Docker multi-stage build for Go binary — needed for CI, decide during Phase 1
  • MySQL schema for pbs_system_metrics and pbs_system_metrics_hourly — design during Phase 1
  • Apache max_workers value — needed to calculate percentage for Layer 3 rules
  • Portainer API auth — how is it currently secured? Needed for Layer 4
  • Claude API prompt engineering — detailed prompt design during Phase 5
  • pbs-hub override page routing — how does it fit into existing pbs-hub Flask app?
  • Google Chat webhook for override commands — new webhook or extend existing?
  • Streamlit deployment — containerized or host-level? Decide during Phase 3

...sent from Jenny & Travis