herbygitea/pbs-projects
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
0de2d1414b
pbs-projects/PBS/Tech/Projects/pbs-security-hardening.md

12 KiB
Raw Blame History

project type status tags created updated path
pbs-security-hardening project-plan active
pbs
docker
traefik
security
crowdsec
authelia
cloudflare
staging
production
automation
n8n
2026-04-02 2026-04-02 PBS/Tech/Projects

PBS Security Hardening - Unified Project Plan

Project Goal

Deploy a layered security stack for PBS infrastructure across staging and production on Linode. Two components:

  1. Crowdsec — Perimeter security with intelligent alerting, replacing manual Wordfence email monitoring
  2. Authelia — SSO with 2FA for all admin/infrastructure services

Approach: Deploy and validate on staging first, then roll to production.

Current Security Baseline

What's in place today

  • Cloudflare: Edge WAF, DNS, CDN, Bot Fight Mode
  • Wordfence: WordPress-level firewall, login lockout (5 attempts), malware scanning
  • 2FA: Enabled on WordPress admin accounts
  • fail2ban: Server-level SSH protection
  • UFW: Firewall with Docker-aware rules in after.rules
  • Cloudflare proxy disabled for n8n and Gitea subdomains

Problems being solved

  • ~20 lockout notification emails per day (noise)
  • No automated intelligent blocking — manual review required
  • Targeted login attempts using real admin usernames (concerning)
  • No SSO — separate credentials for Portainer, Uptime Kuma, n8n, pbs-hub, phpMyAdmin
  • No confidence that attacks aren't slipping through

Architecture Overview

Traffic flow after implementation

Internet → Cloudflare (edge WAF/CDN)
  → Traefik
    → Crowdsec Bouncer (block/allow decision)
      → Authelia (SSO check for admin services)
        → Service (Portainer, n8n, pbs-hub, etc.)
      → WordPress (own auth, Wordfence for malware scanning only)

Tiered alerting model (Crowdsec)

  • Tier 1 — Silent block: Generic bot attempts (admin, test, wp-admin). Automatic. No notification.
  • Tier 2 — Logged, weekly review: Coordinated bot patterns, repeated IPs. Crowdsec handles it.
  • Tier 3 — Immediate alert: Real admin username targeted. Requires WordPress auth log as additional Crowdsec data source. Notification via n8n → Google Chat.
  • Tier 4 — Critical alert: Successful login from unknown IP, failed 2FA on real account, file integrity changes.

Services protected by Authelia

  • Portainer
  • Uptime Kuma
  • n8n
  • pbs-hub
  • phpMyAdmin
  • Traefik dashboard

Services NOT behind Authelia

  • WordPress (uses its own auth + Wordfence + 2FA)

Phase 1: Crowdsec on Staging

Estimated time: 3-4 hours

Step 1.1: Deploy Crowdsec container

  • Add Crowdsec service to staging docker-compose.yml
  • Configure to read Traefik access logs
  • Install WordPress collection (crowdsecurity/wordpress)
  • Install Traefik collection (crowdsecurity/traefik)
  • Verify container starts and parses logs

Step 1.2: Install Traefik Bouncer

  • Add Crowdsec Traefik bouncer plugin to Traefik config
  • Configure bouncer API key
  • Verify bouncer communicates with Crowdsec
  • Test: manually ban an IP via cscli, confirm it gets blocked at Traefik

Step 1.3: Add WordPress auth log source

  • Add WPCode snippet or lightweight plugin to log failed logins with usernames to a file
  • Mount log file into Crowdsec container
  • Configure Crowdsec to parse WordPress auth log as additional datasource
  • Verify Crowdsec sees username-level login data

Step 1.4: Configure community blocklists

  • Register free Crowdsec console account
  • Enroll staging instance
  • Enable community blocklists
  • Verify blocked IPs appear from community intelligence

Step 1.5: Validate on staging

  • Simulate brute force from test IP — confirm auto-block
  • Simulate real username attempt — confirm Tier 3 detection
  • Verify WordPress still accessible for normal logins
  • Verify admin tools still accessible
  • Check cscli metrics for activity
  • Check cscli alerts list for decisions

Phase 2: Crowdsec Alerting via n8n

Estimated time: 2-3 hours

Step 2.1: Crowdsec notification setup

  • Configure Crowdsec HTTP notification plugin
  • Point notifications at n8n webhook endpoint
  • Define alert profiles matching tiered model:
    • Tier 1: No notification (silent block)
    • Tier 2: Logged only (weekly digest — future enhancement)
    • Tier 3+: Immediate webhook to n8n

Step 2.2: n8n alerting workflow

  • Create n8n workflow: Crowdsec webhook → parse alert → route by severity
  • Tier 3 alerts → Google Chat webadmin space via Notify Travis sub-workflow
  • Tier 4 alerts → Google Chat webadmin space with urgent flag
  • Include: IP, reason, username (if available), timestamp, action taken
  • Test end-to-end: trigger alert → verify Google Chat notification

Step 2.3: Validate alerting

  • Confirm Tier 1 produces no notification
  • Confirm Tier 3 sends Google Chat alert with username info
  • Confirm Tier 4 sends urgent alert
  • Verify no false positives from normal browsing

Phase 3: Authelia on Staging

Estimated time: 3-4 hours

Step 3.1: Directory and secrets setup

  • Create directory structure on staging: /opt/docker/authelia/config/
  • Generate secrets: JWT, session, storage encryption, OIDC HMAC
  • Generate password hashes for Travis and Jenny (argon2id)
  • Create users_database.yml with both accounts

Step 3.2: Authelia configuration

  • Write configuration.yml:
    • Session domain: staging domain
    • Default policy: two_factor
    • Storage: SQLite (local file)
    • Notifier: filesystem (for staging — SMTP for production later)
    • TOTP settings: SHA1, 6 digits, 30 second period
  • Define access control rules:
    • Portainer, Uptime Kuma, n8n, pbs-hub, phpMyAdmin, Traefik dashboard → two_factor
    • WordPress and public routes → bypass

Step 3.3: Docker and Traefik integration

  • Create Authelia docker-compose.yml
  • Add Authelia to Traefik network
  • Configure Traefik forwardAuth middleware pointing to Authelia
  • Add middleware labels to all protected services
  • Leave WordPress compose labels unchanged

Step 3.4: Deploy and test

  • Start Authelia container
  • Navigate to a protected service — verify redirect to Authelia login
  • Login with Travis account — verify redirect back to service
  • Register TOTP device
  • Verify 2FA prompt works
  • Test SSO: login once, access other protected services without re-auth
  • Verify WordPress login is unaffected
  • Verify public site is unaffected

Phase 4: Integration Testing on Staging

Estimated time: 2 hours

Step 4.1: Crowdsec + Authelia together

  • Verify Crowdsec bouncer doesn't interfere with Authelia redirects
  • Verify blocked IPs get Crowdsec block page, not Authelia login
  • Test: brute force Authelia login → Crowdsec detects and blocks
  • Verify request flow: Cloudflare → Traefik → Crowdsec → Authelia → Service

Step 4.2: Full stack validation

  • All protected services accessible after Authelia login + 2FA
  • WordPress admin login works independently
  • Public site unaffected
  • Crowdsec blocking bots silently
  • Tier 3/4 alerts arriving in Google Chat
  • No performance degradation

Phase 5: Roll to Production

Estimated time: 2-3 hours

Step 5.1: Pre-production prep

  • Document all staging config that worked
  • Generate fresh secrets for production (never reuse staging secrets)
  • Update Authelia session domain to production domain
  • Switch Authelia notifier from filesystem to SMTP (Google Workspace)
  • Update Crowdsec webhook URL to production n8n

Step 5.2: Deploy Crowdsec to production

  • Add Crowdsec to production docker-compose.yml
  • Add Traefik bouncer plugin
  • Add WordPress auth log source
  • Enroll in Crowdsec console
  • Verify blocking and alerting work

Step 5.3: Deploy Authelia to production

  • Add Authelia to production docker-compose.yml
  • Add forwardAuth middleware to all protected services
  • Verify SSO and 2FA work
  • Register TOTP devices for Travis and Jenny

Step 5.4: Post-deployment

  • Monitor for 48 hours — watch for false positives
  • Tune Crowdsec scenarios if needed
  • Verify Wordfence lockout emails have dropped significantly
  • Demote Wordfence to malware-scanning-only role
  • Consider disabling Wordfence firewall features (now redundant)

Phase 6: Codify in Ansible

Estimated time: 3-4 hours

Step 6.1: Crowdsec Ansible role

  • Create Crowdsec tasks with Ansible tags (--tags crowdsec)
  • Template docker-compose.yml with variables
  • Template Crowdsec acquis.yaml (log sources)
  • Store bouncer API key in ansible-vault
  • Store console enrollment key in ansible-vault

Step 6.2: Authelia Ansible role

  • Create Authelia tasks with Ansible tags (--tags authelia)
  • Template configuration.yml with variables
  • Template users_database.yml
  • Store all secrets in ansible-vault
  • Add forwardAuth middleware labels to protected service templates

Step 6.3: Validate

  • Destroy staging and rebuild from Ansible
  • Verify Crowdsec + Authelia come up correctly
  • Verify alerting works after rebuild
  • Verify SSO works after rebuild

Wordfence Transition Plan

After Crowdsec is stable on production:

  • Disable Wordfence firewall features (redundant with Crowdsec)
  • Keep Wordfence for: scheduled malware scans (monthly), file integrity checks
  • Disable Wordfence login protection (Crowdsec handles this now)
  • Disable lockout email notifications (the whole point!)
  • Document remaining Wordfence role in tech wiki
  • Long-term: when WordPress is replaced, Wordfence goes away entirely

Security Maintenance Cadence

After full deployment:

Daily (glance, 1 min)

  • Check Uptime Kuma — green means good
  • Respond to any Tier 3/4 Google Chat alerts

Weekly (10 min)

  • Review Crowdsec console dashboard for trends
  • Check cscli metrics for volume patterns
  • Update WordPress plugins on staging, then production

Monthly (30 min)

  • Run Wordfence malware scan
  • Review WordPress user accounts
  • Check Docker image updates (Traefik, WordPress, Crowdsec, Authelia)
  • Review Cloudflare analytics
  • Spot-check Crowdsec community blocklist enrollment

Troubleshooting

Crowdsec not blocking

# Check if bouncer is connected
cscli bouncers list

# Check if decisions exist
cscli decisions list

# Check log parsing
cscli metrics

# Manually test a ban
cscli decisions add --ip 1.2.3.4 --reason "test ban" --duration 5m

Authelia not redirecting

# Check Authelia logs
docker logs authelia

# Verify Traefik middleware
docker exec traefik wget -qO- http://localhost:8080/api/http/middlewares

# Check forwardAuth is reaching Authelia
docker logs authelia | grep "authorization"

False positives

# Whitelist your own IP
cscli decisions delete --ip YOUR_IP
cscli parsers install crowdsecurity/whitelists
# Add your IP to /etc/crowdsec/parsers/s02-enrich/whitelists.yaml

Locked out of Authelia

# Reset user password
docker exec authelia authelia hash-password -- 'newpassword'
# Update users_database.yml with new hash
docker restart authelia

Resources

Crowdsec

Authelia

Traefik

Success Criteria

  • Wordfence lockout emails drop to near zero
  • Crowdsec silently blocking bot traffic at Traefik layer
  • Tier 3/4 alerts arriving in Google Chat when real threats detected
  • All admin services behind Authelia SSO with 2FA
  • WordPress auth unaffected
  • Public site unaffected
  • Everything codified in Ansible and reproducible
  • Daily security overhead reduced from 15+ minutes to under 1 minute

Last Updated: April 2, 2026 Maintained by: Travis Project: Plant Based Southerner Infrastructure

...sent from Jenny & Travis