--- project: pbs-security-hardening type: project-plan status: active tags: - pbs - docker - traefik - security - crowdsec - authelia - cloudflare - staging - production - automation - n8n created: 2026-04-02 updated: 2026-04-02 path: PBS/Tech/Projects --- # PBS Security Hardening - Unified Project Plan ## Project Goal Deploy a layered security stack for PBS infrastructure across staging and production on Linode. Two components: 1. **Crowdsec** — Perimeter security with intelligent alerting, replacing manual Wordfence email monitoring 2. **Authelia** — SSO with 2FA for all admin/infrastructure services **Approach:** Deploy and validate on staging first, then roll to production. --- ## Current Security Baseline ### What's in place today - Cloudflare: Edge WAF, DNS, CDN, Bot Fight Mode - Wordfence: WordPress-level firewall, login lockout (5 attempts), malware scanning - 2FA: Enabled on WordPress admin accounts - fail2ban: Server-level SSH protection - UFW: Firewall with Docker-aware rules in after.rules - Cloudflare proxy disabled for n8n and Gitea subdomains ### Problems being solved - ~20 lockout notification emails per day (noise) - No automated intelligent blocking — manual review required - Targeted login attempts using real admin usernames (concerning) - No SSO — separate credentials for Portainer, Uptime Kuma, n8n, pbs-hub, phpMyAdmin - No confidence that attacks aren't slipping through --- ## Architecture Overview ### Traffic flow after implementation ``` Internet → Cloudflare (edge WAF/CDN) → Traefik → Crowdsec Bouncer (block/allow decision) → Authelia (SSO check for admin services) → Service (Portainer, n8n, pbs-hub, etc.) → WordPress (own auth, Wordfence for malware scanning only) ``` ### Tiered alerting model (Crowdsec) - **Tier 1 — Silent block:** Generic bot attempts (admin, test, wp-admin). Automatic. No notification. - **Tier 2 — Logged, weekly review:** Coordinated bot patterns, repeated IPs. Crowdsec handles it. - **Tier 3 — Immediate alert:** Real admin username targeted. Requires WordPress auth log as additional Crowdsec data source. Notification via n8n → Google Chat. - **Tier 4 — Critical alert:** Successful login from unknown IP, failed 2FA on real account, file integrity changes. ### Services protected by Authelia - Portainer - Uptime Kuma - n8n - pbs-hub - phpMyAdmin - Traefik dashboard ### Services NOT behind Authelia - WordPress (uses its own auth + Wordfence + 2FA) --- ## Phase 1: Crowdsec on Staging ### Estimated time: 3-4 hours ### Step 1.1: Deploy Crowdsec container - [ ] Add Crowdsec service to staging docker-compose.yml - [ ] Configure to read Traefik access logs - [ ] Install WordPress collection (crowdsecurity/wordpress) - [ ] Install Traefik collection (crowdsecurity/traefik) - [ ] Verify container starts and parses logs ### Step 1.2: Install Traefik Bouncer - [ ] Add Crowdsec Traefik bouncer plugin to Traefik config - [ ] Configure bouncer API key - [ ] Verify bouncer communicates with Crowdsec - [ ] Test: manually ban an IP via `cscli`, confirm it gets blocked at Traefik ### Step 1.3: Add WordPress auth log source - [ ] Add WPCode snippet or lightweight plugin to log failed logins with usernames to a file - [ ] Mount log file into Crowdsec container - [ ] Configure Crowdsec to parse WordPress auth log as additional datasource - [ ] Verify Crowdsec sees username-level login data ### Step 1.4: Configure community blocklists - [ ] Register free Crowdsec console account - [ ] Enroll staging instance - [ ] Enable community blocklists - [ ] Verify blocked IPs appear from community intelligence ### Step 1.5: Validate on staging - [ ] Simulate brute force from test IP — confirm auto-block - [ ] Simulate real username attempt — confirm Tier 3 detection - [ ] Verify WordPress still accessible for normal logins - [ ] Verify admin tools still accessible - [ ] Check `cscli metrics` for activity - [ ] Check `cscli alerts list` for decisions --- ## Phase 2: Crowdsec Alerting via n8n ### Estimated time: 2-3 hours ### Step 2.1: Crowdsec notification setup - [ ] Configure Crowdsec HTTP notification plugin - [ ] Point notifications at n8n webhook endpoint - [ ] Define alert profiles matching tiered model: - Tier 1: No notification (silent block) - Tier 2: Logged only (weekly digest — future enhancement) - Tier 3+: Immediate webhook to n8n ### Step 2.2: n8n alerting workflow - [ ] Create n8n workflow: Crowdsec webhook → parse alert → route by severity - [ ] Tier 3 alerts → Google Chat `webadmin` space via Notify Travis sub-workflow - [ ] Tier 4 alerts → Google Chat `webadmin` space with urgent flag - [ ] Include: IP, reason, username (if available), timestamp, action taken - [ ] Test end-to-end: trigger alert → verify Google Chat notification ### Step 2.3: Validate alerting - [ ] Confirm Tier 1 produces no notification - [ ] Confirm Tier 3 sends Google Chat alert with username info - [ ] Confirm Tier 4 sends urgent alert - [ ] Verify no false positives from normal browsing --- ## Phase 3: Authelia on Staging ### Estimated time: 3-4 hours ### Step 3.1: Directory and secrets setup - [ ] Create directory structure on staging: `/opt/docker/authelia/config/` - [ ] Generate secrets: JWT, session, storage encryption, OIDC HMAC - [ ] Generate password hashes for Travis and Jenny (argon2id) - [ ] Create users_database.yml with both accounts ### Step 3.2: Authelia configuration - [ ] Write configuration.yml: - Session domain: staging domain - Default policy: two_factor - Storage: SQLite (local file) - Notifier: filesystem (for staging — SMTP for production later) - TOTP settings: SHA1, 6 digits, 30 second period - [ ] Define access control rules: - Portainer, Uptime Kuma, n8n, pbs-hub, phpMyAdmin, Traefik dashboard → two_factor - WordPress and public routes → bypass ### Step 3.3: Docker and Traefik integration - [ ] Create Authelia docker-compose.yml - [ ] Add Authelia to Traefik network - [ ] Configure Traefik forwardAuth middleware pointing to Authelia - [ ] Add middleware labels to all protected services - [ ] Leave WordPress compose labels unchanged ### Step 3.4: Deploy and test - [ ] Start Authelia container - [ ] Navigate to a protected service — verify redirect to Authelia login - [ ] Login with Travis account — verify redirect back to service - [ ] Register TOTP device - [ ] Verify 2FA prompt works - [ ] Test SSO: login once, access other protected services without re-auth - [ ] Verify WordPress login is unaffected - [ ] Verify public site is unaffected --- ## Phase 4: Integration Testing on Staging ### Estimated time: 2 hours ### Step 4.1: Crowdsec + Authelia together - [ ] Verify Crowdsec bouncer doesn't interfere with Authelia redirects - [ ] Verify blocked IPs get Crowdsec block page, not Authelia login - [ ] Test: brute force Authelia login → Crowdsec detects and blocks - [ ] Verify request flow: Cloudflare → Traefik → Crowdsec → Authelia → Service ### Step 4.2: Full stack validation - [ ] All protected services accessible after Authelia login + 2FA - [ ] WordPress admin login works independently - [ ] Public site unaffected - [ ] Crowdsec blocking bots silently - [ ] Tier 3/4 alerts arriving in Google Chat - [ ] No performance degradation --- ## Phase 5: Roll to Production ### Estimated time: 2-3 hours ### Step 5.1: Pre-production prep - [ ] Document all staging config that worked - [ ] Generate fresh secrets for production (never reuse staging secrets) - [ ] Update Authelia session domain to production domain - [ ] Switch Authelia notifier from filesystem to SMTP (Google Workspace) - [ ] Update Crowdsec webhook URL to production n8n ### Step 5.2: Deploy Crowdsec to production - [ ] Add Crowdsec to production docker-compose.yml - [ ] Add Traefik bouncer plugin - [ ] Add WordPress auth log source - [ ] Enroll in Crowdsec console - [ ] Verify blocking and alerting work ### Step 5.3: Deploy Authelia to production - [ ] Add Authelia to production docker-compose.yml - [ ] Add forwardAuth middleware to all protected services - [ ] Verify SSO and 2FA work - [ ] Register TOTP devices for Travis and Jenny ### Step 5.4: Post-deployment - [ ] Monitor for 48 hours — watch for false positives - [ ] Tune Crowdsec scenarios if needed - [ ] Verify Wordfence lockout emails have dropped significantly - [ ] Demote Wordfence to malware-scanning-only role - [ ] Consider disabling Wordfence firewall features (now redundant) --- ## Phase 6: Codify in Ansible ### Estimated time: 3-4 hours ### Step 6.1: Crowdsec Ansible role - [ ] Create Crowdsec tasks with Ansible tags (`--tags crowdsec`) - [ ] Template docker-compose.yml with variables - [ ] Template Crowdsec acquis.yaml (log sources) - [ ] Store bouncer API key in ansible-vault - [ ] Store console enrollment key in ansible-vault ### Step 6.2: Authelia Ansible role - [ ] Create Authelia tasks with Ansible tags (`--tags authelia`) - [ ] Template configuration.yml with variables - [ ] Template users_database.yml - [ ] Store all secrets in ansible-vault - [ ] Add forwardAuth middleware labels to protected service templates ### Step 6.3: Validate - [ ] Destroy staging and rebuild from Ansible - [ ] Verify Crowdsec + Authelia come up correctly - [ ] Verify alerting works after rebuild - [ ] Verify SSO works after rebuild --- ## Wordfence Transition Plan After Crowdsec is stable on production: - [ ] Disable Wordfence firewall features (redundant with Crowdsec) - [ ] Keep Wordfence for: scheduled malware scans (monthly), file integrity checks - [ ] Disable Wordfence login protection (Crowdsec handles this now) - [ ] Disable lockout email notifications (the whole point!) - [ ] Document remaining Wordfence role in tech wiki - [ ] Long-term: when WordPress is replaced, Wordfence goes away entirely --- ## Security Maintenance Cadence After full deployment: ### Daily (glance, 1 min) - Check Uptime Kuma — green means good - Respond to any Tier 3/4 Google Chat alerts ### Weekly (10 min) - Review Crowdsec console dashboard for trends - Check `cscli metrics` for volume patterns - Update WordPress plugins on staging, then production ### Monthly (30 min) - Run Wordfence malware scan - Review WordPress user accounts - Check Docker image updates (Traefik, WordPress, Crowdsec, Authelia) - Review Cloudflare analytics - Spot-check Crowdsec community blocklist enrollment --- ## Troubleshooting ### Crowdsec not blocking ```bash # Check if bouncer is connected cscli bouncers list # Check if decisions exist cscli decisions list # Check log parsing cscli metrics # Manually test a ban cscli decisions add --ip 1.2.3.4 --reason "test ban" --duration 5m ``` ### Authelia not redirecting ```bash # Check Authelia logs docker logs authelia # Verify Traefik middleware docker exec traefik wget -qO- http://localhost:8080/api/http/middlewares # Check forwardAuth is reaching Authelia docker logs authelia | grep "authorization" ``` ### False positives ```bash # Whitelist your own IP cscli decisions delete --ip YOUR_IP cscli parsers install crowdsecurity/whitelists # Add your IP to /etc/crowdsec/parsers/s02-enrich/whitelists.yaml ``` ### Locked out of Authelia ```bash # Reset user password docker exec authelia authelia hash-password -- 'newpassword' # Update users_database.yml with new hash docker restart authelia ``` --- ## Resources ### Crowdsec - Official docs: https://docs.crowdsec.net/ - Traefik bouncer: https://github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin - WordPress collection: https://hub.crowdsec.net/author/crowdsecurity/collections/wordpress - Console: https://app.crowdsec.net/ ### Authelia - Official docs: https://www.authelia.com/ - Configuration reference: https://www.authelia.com/configuration/prologue/introduction/ - Traefik integration: https://www.authelia.com/integration/proxies/traefik/ - File-based users: https://www.authelia.com/reference/guides/passwords/ ### Traefik - ForwardAuth middleware: https://doc.traefik.io/traefik/middlewares/http/forwardauth/ - Plugin catalog: https://plugins.traefik.io/ --- ## Success Criteria - [ ] Wordfence lockout emails drop to near zero - [ ] Crowdsec silently blocking bot traffic at Traefik layer - [ ] Tier 3/4 alerts arriving in Google Chat when real threats detected - [ ] All admin services behind Authelia SSO with 2FA - [ ] WordPress auth unaffected - [ ] Public site unaffected - [ ] Everything codified in Ansible and reproducible - [ ] Daily security overhead reduced from 15+ minutes to under 1 minute --- *Last Updated: April 2, 2026* *Maintained by: Travis* *Project: Plant Based Southerner Infrastructure* ...sent from Jenny & Travis