petal-power/wiki-vault
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
12a57499cb
wiki-vault/Sources/Homelab/pbs-security-hardening.md
Travis Herbranson 34a268d8dc migration: copy 62 notes from pbs-projects and homelab-projects 
Two-layer structure: Sources (raw notes) + Wiki (compile output)
Four domains: Dev (40), Venture (3), Homelab (23), Reference (0)
Includes CLAUDE.md spec, index pages at all levels, compile log

Co-Authored-By: Lovebug <lovebug@herbylab.dev>
2026-05-08 17:21:13 -04:00

12 KiB
Raw Blame History

created path project status tags type updated
2026-04-02 Sources/Homelab pbs-security-hardening active
pbs
docker
traefik
security
crowdsec
authelia
cloudflare
staging
production
automation
n8n
project-plan 2026-04-02

PBS Security Hardening - Unified Project Plan

Project Goal

Deploy a layered security stack for PBS infrastructure across staging and production on Linode. Two components:

  1. Crowdsec — Perimeter security with intelligent alerting, replacing manual Wordfence email monitoring
  2. Authelia — SSO with 2FA for all admin/infrastructure services

Approach: Deploy and validate on staging first, then roll to production.

Current Security Baseline

What's in place today

  • Cloudflare: Edge WAF, DNS, CDN, Bot Fight Mode
  • Wordfence: WordPress-level firewall, login lockout (5 attempts), malware scanning
  • 2FA: Enabled on WordPress admin accounts
  • fail2ban: Server-level SSH protection
  • UFW: Firewall with Docker-aware rules in after.rules
  • Cloudflare proxy disabled for n8n and Gitea subdomains

Problems being solved

  • ~20 lockout notification emails per day (noise)
  • No automated intelligent blocking — manual review required
  • Targeted login attempts using real admin usernames (concerning)
  • No SSO — separate credentials for Portainer, Uptime Kuma, n8n, pbs-hub, phpMyAdmin
  • No confidence that attacks aren't slipping through

Architecture Overview

Traffic flow after implementation

Internet → Cloudflare (edge WAF/CDN) → Traefik → Crowdsec Bouncer (block/allow decision) → Authelia (SSO check for admin services) → Service (Portainer, n8n, pbs-hub, etc.) → WordPress (own auth, Wordfence for malware scanning only)

Tiered alerting model (Crowdsec)

  • Tier 1 — Silent block: Generic bot attempts (admin, test, wp-admin). Automatic. No notification.
  • Tier 2 — Logged, weekly review: Coordinated bot patterns, repeated IPs. Crowdsec handles it.
  • Tier 3 — Immediate alert: Real admin username targeted. Requires WordPress auth log as additional Crowdsec data source. Notification via n8n → Google Chat.
  • Tier 4 — Critical alert: Successful login from unknown IP, failed 2FA on real account, file integrity changes.

Services protected by Authelia

  • Portainer
  • Uptime Kuma
  • n8n
  • pbs-hub
  • phpMyAdmin
  • Traefik dashboard

Services NOT behind Authelia

  • WordPress (uses its own auth + Wordfence + 2FA)

Phase 1: Crowdsec on Staging

Estimated time: 3-4 hours

Step 1.1: Deploy Crowdsec container

  • Add Crowdsec service to staging docker-compose.yml
  • Configure to read Traefik access logs
  • Install WordPress collection (crowdsecurity/wordpress)
  • Install Traefik collection (crowdsecurity/traefik)
  • Verify container starts and parses logs

Step 1.2: Install Traefik Bouncer

  • Add Crowdsec Traefik bouncer plugin to Traefik config
  • Configure bouncer API key
  • Verify bouncer communicates with Crowdsec
  • Test: manually ban an IP via cscli, confirm it gets blocked at Traefik

Step 1.3: Add WordPress auth log source

  • Add WPCode snippet or lightweight plugin to log failed logins with usernames to a file
  • Mount log file into Crowdsec container
  • Configure Crowdsec to parse WordPress auth log as additional datasource
  • Verify Crowdsec sees username-level login data

Step 1.4: Configure community blocklists

  • Register free Crowdsec console account
  • Enroll staging instance
  • Enable community blocklists
  • Verify blocked IPs appear from community intelligence

Step 1.5: Validate on staging

  • Simulate brute force from test IP — confirm auto-block
  • Simulate real username attempt — confirm Tier 3 detection
  • Verify WordPress still accessible for normal logins
  • Verify admin tools still accessible
  • Check cscli metrics for activity
  • Check cscli alerts list for decisions

Phase 2: Crowdsec Alerting via n8n

Estimated time: 2-3 hours

Step 2.1: Crowdsec notification setup

  • Configure Crowdsec HTTP notification plugin
  • Point notifications at n8n webhook endpoint
  • Define alert profiles matching tiered model:
    • Tier 1: No notification (silent block)
    • Tier 2: Logged only (weekly digest — future enhancement)
    • Tier 3+: Immediate webhook to n8n

Step 2.2: n8n alerting workflow

  • Create n8n workflow: Crowdsec webhook → parse alert → route by severity
  • Tier 3 alerts → Google Chat webadmin space via Notify Travis sub-workflow
  • Tier 4 alerts → Google Chat webadmin space with urgent flag
  • Include: IP, reason, username (if available), timestamp, action taken
  • Test end-to-end: trigger alert → verify Google Chat notification

Step 2.3: Validate alerting

  • Confirm Tier 1 produces no notification
  • Confirm Tier 3 sends Google Chat alert with username info
  • Confirm Tier 4 sends urgent alert
  • Verify no false positives from normal browsing

Phase 3: Authelia on Staging

Estimated time: 3-4 hours

Step 3.1: Directory and secrets setup

  • Create directory structure on staging: /opt/docker/authelia/config/
  • Generate secrets: JWT, session, storage encryption, OIDC HMAC
  • Generate password hashes for Travis and Jenny (argon2id)
  • Create users_database.yml with both accounts

Step 3.2: Authelia configuration

  • Write configuration.yml:
    • Session domain: staging domain
    • Default policy: two_factor
    • Storage: SQLite (local file)
    • Notifier: filesystem (for staging — SMTP for production later)
    • TOTP settings: SHA1, 6 digits, 30 second period
  • Define access control rules:
    • Portainer, Uptime Kuma, n8n, pbs-hub, phpMyAdmin, Traefik dashboard → two_factor
    • WordPress and public routes → bypass

Step 3.3: Docker and Traefik integration

  • Create Authelia docker-compose.yml
  • Add Authelia to Traefik network
  • Configure Traefik forwardAuth middleware pointing to Authelia
  • Add middleware labels to all protected services
  • Leave WordPress compose labels unchanged

Step 3.4: Deploy and test

  • Start Authelia container
  • Navigate to a protected service — verify redirect to Authelia login
  • Login with Travis account — verify redirect back to service
  • Register TOTP device
  • Verify 2FA prompt works
  • Test SSO: login once, access other protected services without re-auth
  • Verify WordPress login is unaffected
  • Verify public site is unaffected

Phase 4: Integration Testing on Staging

Estimated time: 2 hours

Step 4.1: Crowdsec + Authelia together

  • Verify Crowdsec bouncer doesn't interfere with Authelia redirects
  • Verify blocked IPs get Crowdsec block page, not Authelia login
  • Test: brute force Authelia login → Crowdsec detects and blocks
  • Verify request flow: Cloudflare → Traefik → Crowdsec → Authelia → Service

Step 4.2: Full stack validation

  • All protected services accessible after Authelia login + 2FA
  • WordPress admin login works independently
  • Public site unaffected
  • Crowdsec blocking bots silently
  • Tier 3/4 alerts arriving in Google Chat
  • No performance degradation

Phase 5: Roll to Production

Estimated time: 2-3 hours

Step 5.1: Pre-production prep

  • Document all staging config that worked
  • Generate fresh secrets for production (never reuse staging secrets)
  • Update Authelia session domain to production domain
  • Switch Authelia notifier from filesystem to SMTP (Google Workspace)
  • Update Crowdsec webhook URL to production n8n

Step 5.2: Deploy Crowdsec to production

  • Add Crowdsec to production docker-compose.yml
  • Add Traefik bouncer plugin
  • Add WordPress auth log source
  • Enroll in Crowdsec console
  • Verify blocking and alerting work

Step 5.3: Deploy Authelia to production

  • Add Authelia to production docker-compose.yml
  • Add forwardAuth middleware to all protected services
  • Verify SSO and 2FA work
  • Register TOTP devices for Travis and Jenny

Step 5.4: Post-deployment

  • Monitor for 48 hours — watch for false positives
  • Tune Crowdsec scenarios if needed
  • Verify Wordfence lockout emails have dropped significantly
  • Demote Wordfence to malware-scanning-only role
  • Consider disabling Wordfence firewall features (now redundant)

Phase 6: Codify in Ansible

Estimated time: 3-4 hours

Step 6.1: Crowdsec Ansible role

  • Create Crowdsec tasks with Ansible tags (--tags crowdsec)
  • Template docker-compose.yml with variables
  • Template Crowdsec acquis.yaml (log sources)
  • Store bouncer API key in ansible-vault
  • Store console enrollment key in ansible-vault

Step 6.2: Authelia Ansible role

  • Create Authelia tasks with Ansible tags (--tags authelia)
  • Template configuration.yml with variables
  • Template users_database.yml
  • Store all secrets in ansible-vault
  • Add forwardAuth middleware labels to protected service templates

Step 6.3: Validate

  • Destroy staging and rebuild from Ansible
  • Verify Crowdsec + Authelia come up correctly
  • Verify alerting works after rebuild
  • Verify SSO works after rebuild

Wordfence Transition Plan

After Crowdsec is stable on production:

  • Disable Wordfence firewall features (redundant with Crowdsec)
  • Keep Wordfence for: scheduled malware scans (monthly), file integrity checks
  • Disable Wordfence login protection (Crowdsec handles this now)
  • Disable lockout email notifications (the whole point!)
  • Document remaining Wordfence role in tech wiki
  • Long-term: when WordPress is replaced, Wordfence goes away entirely

Security Maintenance Cadence

After full deployment:

Daily (glance, 1 min)

  • Check Uptime Kuma — green means good
  • Respond to any Tier 3/4 Google Chat alerts

Weekly (10 min)

  • Review Crowdsec console dashboard for trends
  • Check cscli metrics for volume patterns
  • Update WordPress plugins on staging, then production

Monthly (30 min)

  • Run Wordfence malware scan
  • Review WordPress user accounts
  • Check Docker image updates (Traefik, WordPress, Crowdsec, Authelia)
  • Review Cloudflare analytics
  • Spot-check Crowdsec community blocklist enrollment

Troubleshooting

Crowdsec not blocking

bash

Check if bouncer is connected

cscli bouncers list

Check if decisions exist

cscli decisions list

Check log parsing

cscli metrics

Manually test a ban

cscli decisions add --ip 1.2.3.4 --reason "test ban" --duration 5m

Authelia not redirecting

bash

Check Authelia logs

docker logs authelia

Verify Traefik middleware

docker exec traefik wget -qO- http://localhost:8080/api/http/middlewares

Check forwardAuth is reaching Authelia

docker logs authelia | grep "authorization"

False positives

bash

Whitelist your own IP

cscli decisions delete --ip YOUR_IP cscli parsers install crowdsecurity/whitelists

Add your IP to /etc/crowdsec/parsers/s02-enrich/whitelists.yaml

Locked out of Authelia

bash

Reset user password

docker exec authelia authelia hash-password -- 'newpassword'

Update users_database.yml with new hash

docker restart authelia

Resources

Crowdsec

Authelia

Traefik

Success Criteria

  • Wordfence lockout emails drop to near zero
  • Crowdsec silently blocking bot traffic at Traefik layer
  • Tier 3/4 alerts arriving in Google Chat when real threats detected
  • All admin services behind Authelia SSO with 2FA
  • WordPress auth unaffected
  • Public site unaffected
  • Everything codified in Ansible and reproducible
  • Daily security overhead reduced from 15+ minutes to under 1 minute

Last Updated: April 2, 2026 Maintained by: Travis Project: Plant Based Southerner Infrastructure

...sent from Jenny & Travis