From c44d2428d019d5e21f1ecd394bee2257527f2d70 Mon Sep 17 00:00:00 2001 From: herbygitea Date: Fri, 3 Apr 2026 01:00:08 +0000 Subject: [PATCH] Create pbs-security-hardening.md via n8n --- PBS/Tech/Projects/pbs-security-hardening.md | 410 ++++++++++++++++++++ 1 file changed, 410 insertions(+) create mode 100644 PBS/Tech/Projects/pbs-security-hardening.md diff --git a/PBS/Tech/Projects/pbs-security-hardening.md b/PBS/Tech/Projects/pbs-security-hardening.md new file mode 100644 index 0000000..3fb8ebc --- /dev/null +++ b/PBS/Tech/Projects/pbs-security-hardening.md @@ -0,0 +1,410 @@ +--- +project: pbs-security-hardening +type: project-plan +status: active +tags: + - pbs + - docker + - traefik + - security + - crowdsec + - authelia + - cloudflare + - staging + - production + - automation + - n8n +created: 2026-04-02 +updated: 2026-04-02 +path: PBS/Tech/Projects +--- + +# PBS Security Hardening - Unified Project Plan + +## Project Goal + +Deploy a layered security stack for PBS infrastructure across staging and +production on Linode. Two components: + +1. **Crowdsec** — Perimeter security with intelligent alerting, replacing +manual Wordfence email monitoring +2. **Authelia** — SSO with 2FA for all admin/infrastructure services + +**Approach:** Deploy and validate on staging first, then roll to production. + +--- + +## Current Security Baseline + +### What's in place today +- Cloudflare: Edge WAF, DNS, CDN, Bot Fight Mode +- Wordfence: WordPress-level firewall, login lockout (5 attempts), malware +scanning +- 2FA: Enabled on WordPress admin accounts +- fail2ban: Server-level SSH protection +- UFW: Firewall with Docker-aware rules in after.rules +- Cloudflare proxy disabled for n8n and Gitea subdomains + +### Problems being solved +- ~20 lockout notification emails per day (noise) +- No automated intelligent blocking — manual review required +- Targeted login attempts using real admin usernames (concerning) +- No SSO — separate credentials for Portainer, Uptime Kuma, n8n, pbs-hub, +phpMyAdmin +- No confidence that attacks aren't slipping through + +--- + +## Architecture Overview + +### Traffic flow after implementation +``` +Internet → Cloudflare (edge WAF/CDN) + → Traefik + → Crowdsec Bouncer (block/allow decision) + → Authelia (SSO check for admin services) + → Service (Portainer, n8n, pbs-hub, etc.) + → WordPress (own auth, Wordfence for malware scanning only) +``` + +### Tiered alerting model (Crowdsec) +- **Tier 1 — Silent block:** Generic bot attempts (admin, test, wp-admin). +Automatic. No notification. +- **Tier 2 — Logged, weekly review:** Coordinated bot patterns, repeated +IPs. Crowdsec handles it. +- **Tier 3 — Immediate alert:** Real admin username targeted. Requires +WordPress auth log as additional Crowdsec data source. Notification via n8n +→ Google Chat. +- **Tier 4 — Critical alert:** Successful login from unknown IP, failed 2FA +on real account, file integrity changes. + +### Services protected by Authelia +- Portainer +- Uptime Kuma +- n8n +- pbs-hub +- phpMyAdmin +- Traefik dashboard + +### Services NOT behind Authelia +- WordPress (uses its own auth + Wordfence + 2FA) + +--- + +## Phase 1: Crowdsec on Staging + +### Estimated time: 3-4 hours + +### Step 1.1: Deploy Crowdsec container +- [ ] Add Crowdsec service to staging docker-compose.yml +- [ ] Configure to read Traefik access logs +- [ ] Install WordPress collection (crowdsecurity/wordpress) +- [ ] Install Traefik collection (crowdsecurity/traefik) +- [ ] Verify container starts and parses logs + +### Step 1.2: Install Traefik Bouncer +- [ ] Add Crowdsec Traefik bouncer plugin to Traefik config +- [ ] Configure bouncer API key +- [ ] Verify bouncer communicates with Crowdsec +- [ ] Test: manually ban an IP via `cscli`, confirm it gets blocked at +Traefik + +### Step 1.3: Add WordPress auth log source +- [ ] Add WPCode snippet or lightweight plugin to log failed logins with +usernames to a file +- [ ] Mount log file into Crowdsec container +- [ ] Configure Crowdsec to parse WordPress auth log as additional +datasource +- [ ] Verify Crowdsec sees username-level login data + +### Step 1.4: Configure community blocklists +- [ ] Register free Crowdsec console account +- [ ] Enroll staging instance +- [ ] Enable community blocklists +- [ ] Verify blocked IPs appear from community intelligence + +### Step 1.5: Validate on staging +- [ ] Simulate brute force from test IP — confirm auto-block +- [ ] Simulate real username attempt — confirm Tier 3 detection +- [ ] Verify WordPress still accessible for normal logins +- [ ] Verify admin tools still accessible +- [ ] Check `cscli metrics` for activity +- [ ] Check `cscli alerts list` for decisions + +--- + +## Phase 2: Crowdsec Alerting via n8n + +### Estimated time: 2-3 hours + +### Step 2.1: Crowdsec notification setup +- [ ] Configure Crowdsec HTTP notification plugin +- [ ] Point notifications at n8n webhook endpoint +- [ ] Define alert profiles matching tiered model: + - Tier 1: No notification (silent block) + - Tier 2: Logged only (weekly digest — future enhancement) + - Tier 3+: Immediate webhook to n8n + +### Step 2.2: n8n alerting workflow +- [ ] Create n8n workflow: Crowdsec webhook → parse alert → route by +severity +- [ ] Tier 3 alerts → Google Chat `webadmin` space via Notify Travis +sub-workflow +- [ ] Tier 4 alerts → Google Chat `webadmin` space with urgent flag +- [ ] Include: IP, reason, username (if available), timestamp, action taken +- [ ] Test end-to-end: trigger alert → verify Google Chat notification + +### Step 2.3: Validate alerting +- [ ] Confirm Tier 1 produces no notification +- [ ] Confirm Tier 3 sends Google Chat alert with username info +- [ ] Confirm Tier 4 sends urgent alert +- [ ] Verify no false positives from normal browsing + +--- + +## Phase 3: Authelia on Staging + +### Estimated time: 3-4 hours + +### Step 3.1: Directory and secrets setup +- [ ] Create directory structure on staging: `/opt/docker/authelia/config/` +- [ ] Generate secrets: JWT, session, storage encryption, OIDC HMAC +- [ ] Generate password hashes for Travis and Jenny (argon2id) +- [ ] Create users_database.yml with both accounts + +### Step 3.2: Authelia configuration +- [ ] Write configuration.yml: + - Session domain: staging domain + - Default policy: two_factor + - Storage: SQLite (local file) + - Notifier: filesystem (for staging — SMTP for production later) + - TOTP settings: SHA1, 6 digits, 30 second period +- [ ] Define access control rules: + - Portainer, Uptime Kuma, n8n, pbs-hub, phpMyAdmin, Traefik dashboard → +two_factor + - WordPress and public routes → bypass + +### Step 3.3: Docker and Traefik integration +- [ ] Create Authelia docker-compose.yml +- [ ] Add Authelia to Traefik network +- [ ] Configure Traefik forwardAuth middleware pointing to Authelia +- [ ] Add middleware labels to all protected services +- [ ] Leave WordPress compose labels unchanged + +### Step 3.4: Deploy and test +- [ ] Start Authelia container +- [ ] Navigate to a protected service — verify redirect to Authelia login +- [ ] Login with Travis account — verify redirect back to service +- [ ] Register TOTP device +- [ ] Verify 2FA prompt works +- [ ] Test SSO: login once, access other protected services without re-auth +- [ ] Verify WordPress login is unaffected +- [ ] Verify public site is unaffected + +--- + +## Phase 4: Integration Testing on Staging + +### Estimated time: 2 hours + +### Step 4.1: Crowdsec + Authelia together +- [ ] Verify Crowdsec bouncer doesn't interfere with Authelia redirects +- [ ] Verify blocked IPs get Crowdsec block page, not Authelia login +- [ ] Test: brute force Authelia login → Crowdsec detects and blocks +- [ ] Verify request flow: Cloudflare → Traefik → Crowdsec → Authelia → +Service + +### Step 4.2: Full stack validation +- [ ] All protected services accessible after Authelia login + 2FA +- [ ] WordPress admin login works independently +- [ ] Public site unaffected +- [ ] Crowdsec blocking bots silently +- [ ] Tier 3/4 alerts arriving in Google Chat +- [ ] No performance degradation + +--- + +## Phase 5: Roll to Production + +### Estimated time: 2-3 hours + +### Step 5.1: Pre-production prep +- [ ] Document all staging config that worked +- [ ] Generate fresh secrets for production (never reuse staging secrets) +- [ ] Update Authelia session domain to production domain +- [ ] Switch Authelia notifier from filesystem to SMTP (Google Workspace) +- [ ] Update Crowdsec webhook URL to production n8n + +### Step 5.2: Deploy Crowdsec to production +- [ ] Add Crowdsec to production docker-compose.yml +- [ ] Add Traefik bouncer plugin +- [ ] Add WordPress auth log source +- [ ] Enroll in Crowdsec console +- [ ] Verify blocking and alerting work + +### Step 5.3: Deploy Authelia to production +- [ ] Add Authelia to production docker-compose.yml +- [ ] Add forwardAuth middleware to all protected services +- [ ] Verify SSO and 2FA work +- [ ] Register TOTP devices for Travis and Jenny + +### Step 5.4: Post-deployment +- [ ] Monitor for 48 hours — watch for false positives +- [ ] Tune Crowdsec scenarios if needed +- [ ] Verify Wordfence lockout emails have dropped significantly +- [ ] Demote Wordfence to malware-scanning-only role +- [ ] Consider disabling Wordfence firewall features (now redundant) + +--- + +## Phase 6: Codify in Ansible + +### Estimated time: 3-4 hours + +### Step 6.1: Crowdsec Ansible role +- [ ] Create Crowdsec tasks with Ansible tags (`--tags crowdsec`) +- [ ] Template docker-compose.yml with variables +- [ ] Template Crowdsec acquis.yaml (log sources) +- [ ] Store bouncer API key in ansible-vault +- [ ] Store console enrollment key in ansible-vault + +### Step 6.2: Authelia Ansible role +- [ ] Create Authelia tasks with Ansible tags (`--tags authelia`) +- [ ] Template configuration.yml with variables +- [ ] Template users_database.yml +- [ ] Store all secrets in ansible-vault +- [ ] Add forwardAuth middleware labels to protected service templates + +### Step 6.3: Validate +- [ ] Destroy staging and rebuild from Ansible +- [ ] Verify Crowdsec + Authelia come up correctly +- [ ] Verify alerting works after rebuild +- [ ] Verify SSO works after rebuild + +--- + +## Wordfence Transition Plan + +After Crowdsec is stable on production: + +- [ ] Disable Wordfence firewall features (redundant with Crowdsec) +- [ ] Keep Wordfence for: scheduled malware scans (monthly), file integrity +checks +- [ ] Disable Wordfence login protection (Crowdsec handles this now) +- [ ] Disable lockout email notifications (the whole point!) +- [ ] Document remaining Wordfence role in tech wiki +- [ ] Long-term: when WordPress is replaced, Wordfence goes away entirely + +--- + +## Security Maintenance Cadence + +After full deployment: + +### Daily (glance, 1 min) +- Check Uptime Kuma — green means good +- Respond to any Tier 3/4 Google Chat alerts + +### Weekly (10 min) +- Review Crowdsec console dashboard for trends +- Check `cscli metrics` for volume patterns +- Update WordPress plugins on staging, then production + +### Monthly (30 min) +- Run Wordfence malware scan +- Review WordPress user accounts +- Check Docker image updates (Traefik, WordPress, Crowdsec, Authelia) +- Review Cloudflare analytics +- Spot-check Crowdsec community blocklist enrollment + +--- + +## Troubleshooting + +### Crowdsec not blocking +```bash +# Check if bouncer is connected +cscli bouncers list + +# Check if decisions exist +cscli decisions list + +# Check log parsing +cscli metrics + +# Manually test a ban +cscli decisions add --ip 1.2.3.4 --reason "test ban" --duration 5m +``` + +### Authelia not redirecting +```bash +# Check Authelia logs +docker logs authelia + +# Verify Traefik middleware +docker exec traefik wget -qO- http://localhost:8080/api/http/middlewares + +# Check forwardAuth is reaching Authelia +docker logs authelia | grep "authorization" +``` + +### False positives +```bash +# Whitelist your own IP +cscli decisions delete --ip YOUR_IP +cscli parsers install crowdsecurity/whitelists +# Add your IP to /etc/crowdsec/parsers/s02-enrich/whitelists.yaml +``` + +### Locked out of Authelia +```bash +# Reset user password +docker exec authelia authelia hash-password -- 'newpassword' +# Update users_database.yml with new hash +docker restart authelia +``` + +--- + +## Resources + +### Crowdsec +- Official docs: https://docs.crowdsec.net/ +- Traefik bouncer: +https://github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin +- WordPress collection: +https://hub.crowdsec.net/author/crowdsecurity/collections/wordpress +- Console: https://app.crowdsec.net/ + +### Authelia +- Official docs: https://www.authelia.com/ +- Configuration reference: +https://www.authelia.com/configuration/prologue/introduction/ +- Traefik integration: https://www.authelia.com/integration/proxies/traefik/ +- File-based users: https://www.authelia.com/reference/guides/passwords/ + +### Traefik +- ForwardAuth middleware: +https://doc.traefik.io/traefik/middlewares/http/forwardauth/ +- Plugin catalog: https://plugins.traefik.io/ + +--- + +## Success Criteria + +- [ ] Wordfence lockout emails drop to near zero +- [ ] Crowdsec silently blocking bot traffic at Traefik layer +- [ ] Tier 3/4 alerts arriving in Google Chat when real threats detected +- [ ] All admin services behind Authelia SSO with 2FA +- [ ] WordPress auth unaffected +- [ ] Public site unaffected +- [ ] Everything codified in Ansible and reproducible +- [ ] Daily security overhead reduced from 15+ minutes to under 1 minute + +--- + +*Last Updated: April 2, 2026* +*Maintained by: Travis* +*Project: Plant Based Southerner Infrastructure* + +...sent from Jenny & Travis \ No newline at end of file