382 lines
12 KiB
Markdown
382 lines
12 KiB
Markdown
|
|
---
|
|
project: pbs-security-hardening
|
|
type: project-plan
|
|
status: active
|
|
tags:
|
|
- pbs
|
|
- docker
|
|
- traefik
|
|
- security
|
|
- crowdsec
|
|
- authelia
|
|
- cloudflare
|
|
- staging
|
|
- production
|
|
- automation
|
|
- n8n
|
|
created: 2026-04-02
|
|
updated: 2026-04-02
|
|
path: PBS/Tech/Projects/
|
|
---
|
|
# PBS Security Hardening - Unified Project Plan
|
|
|
|
## Project Goal
|
|
|
|
Deploy a layered security stack for PBS infrastructure across staging and
|
|
production on Linode. Two components:
|
|
|
|
1. **Crowdsec** — Perimeter security with intelligent alerting, replacing
|
|
manual Wordfence email monitoring
|
|
2. **Authelia** — SSO with 2FA for all admin/infrastructure services
|
|
|
|
**Approach:** Deploy and validate on staging first, then roll to production.
|
|
---
|
|
## Current Security Baseline
|
|
|
|
### What's in place today
|
|
- Cloudflare: Edge WAF, DNS, CDN, Bot Fight Mode
|
|
- Wordfence: WordPress-level firewall, login lockout (5 attempts), malware
|
|
scanning
|
|
- 2FA: Enabled on WordPress admin accounts
|
|
- fail2ban: Server-level SSH protection
|
|
- UFW: Firewall with Docker-aware rules in after.rules
|
|
- Cloudflare proxy disabled for n8n and Gitea subdomains
|
|
|
|
### Problems being solved
|
|
- ~20 lockout notification emails per day (noise)
|
|
- No automated intelligent blocking — manual review required
|
|
- Targeted login attempts using real admin usernames (concerning)
|
|
- No SSO — separate credentials for Portainer, Uptime Kuma, n8n, pbs-hub,
|
|
phpMyAdmin
|
|
- No confidence that attacks aren't slipping through
|
|
---
|
|
## Architecture Overview
|
|
|
|
### Traffic flow after implementation
|
|
|
|
Internet → Cloudflare (edge WAF/CDN)
|
|
→ Traefik
|
|
→ Crowdsec Bouncer (block/allow decision)
|
|
→ Authelia (SSO check for admin services)
|
|
→ Service (Portainer, n8n, pbs-hub, etc.)
|
|
→ WordPress (own auth, Wordfence for malware scanning only)
|
|
|
|
|
|
### Tiered alerting model (Crowdsec)
|
|
- **Tier 1 — Silent block:** Generic bot attempts (admin, test, wp-admin).
|
|
Automatic. No notification.
|
|
- **Tier 2 — Logged, weekly review:** Coordinated bot patterns, repeated
|
|
IPs. Crowdsec handles it.
|
|
- **Tier 3 — Immediate alert:** Real admin username targeted. Requires
|
|
WordPress auth log as additional Crowdsec data source. Notification via n8n
|
|
→ Google Chat.
|
|
- **Tier 4 — Critical alert:** Successful login from unknown IP, failed 2FA
|
|
on real account, file integrity changes.
|
|
|
|
### Services protected by Authelia
|
|
- Portainer
|
|
- Uptime Kuma
|
|
- n8n
|
|
- pbs-hub
|
|
- phpMyAdmin
|
|
- Traefik dashboard
|
|
|
|
### Services NOT behind Authelia
|
|
- WordPress (uses its own auth + Wordfence + 2FA)
|
|
---
|
|
## Phase 1: Crowdsec on Staging
|
|
|
|
### Estimated time: 3-4 hours
|
|
|
|
### Step 1.1: Deploy Crowdsec container
|
|
- [ ] Add Crowdsec service to staging docker-compose.yml
|
|
- [ ] Configure to read Traefik access logs
|
|
- [ ] Install WordPress collection (crowdsecurity/wordpress)
|
|
- [ ] Install Traefik collection (crowdsecurity/traefik)
|
|
- [ ] Verify container starts and parses logs
|
|
|
|
### Step 1.2: Install Traefik Bouncer
|
|
- [ ] Add Crowdsec Traefik bouncer plugin to Traefik config
|
|
- [ ] Configure bouncer API key
|
|
- [ ] Verify bouncer communicates with Crowdsec
|
|
- [ ] Test: manually ban an IP via `cscli`, confirm it gets blocked at
|
|
Traefik
|
|
|
|
### Step 1.3: Add WordPress auth log source
|
|
- [ ] Add WPCode snippet or lightweight plugin to log failed logins with
|
|
usernames to a file
|
|
- [ ] Mount log file into Crowdsec container
|
|
- [ ] Configure Crowdsec to parse WordPress auth log as additional
|
|
datasource
|
|
- [ ] Verify Crowdsec sees username-level login data
|
|
|
|
### Step 1.4: Configure community blocklists
|
|
- [ ] Register free Crowdsec console account
|
|
- [ ] Enroll staging instance
|
|
- [ ] Enable community blocklists
|
|
- [ ] Verify blocked IPs appear from community intelligence
|
|
|
|
### Step 1.5: Validate on staging
|
|
- [ ] Simulate brute force from test IP — confirm auto-block
|
|
- [ ] Simulate real username attempt — confirm Tier 3 detection
|
|
- [ ] Verify WordPress still accessible for normal logins
|
|
- [ ] Verify admin tools still accessible
|
|
- [ ] Check `cscli metrics` for activity
|
|
- [ ] Check `cscli alerts list` for decisions
|
|
---
|
|
## Phase 2: Crowdsec Alerting via n8n
|
|
|
|
### Estimated time: 2-3 hours
|
|
|
|
### Step 2.1: Crowdsec notification setup
|
|
- [ ] Configure Crowdsec HTTP notification plugin
|
|
- [ ] Point notifications at n8n webhook endpoint
|
|
- [ ] Define alert profiles matching tiered model:
|
|
- Tier 1: No notification (silent block)
|
|
- Tier 2: Logged only (weekly digest — future enhancement)
|
|
- Tier 3+: Immediate webhook to n8n
|
|
|
|
### Step 2.2: n8n alerting workflow
|
|
- [ ] Create n8n workflow: Crowdsec webhook → parse alert → route by
|
|
severity
|
|
- [ ] Tier 3 alerts → Google Chat `webadmin` space via Notify Travis
|
|
sub-workflow
|
|
- [ ] Tier 4 alerts → Google Chat `webadmin` space with urgent flag
|
|
- [ ] Include: IP, reason, username (if available), timestamp, action taken
|
|
- [ ] Test end-to-end: trigger alert → verify Google Chat notification
|
|
|
|
### Step 2.3: Validate alerting
|
|
- [ ] Confirm Tier 1 produces no notification
|
|
- [ ] Confirm Tier 3 sends Google Chat alert with username info
|
|
- [ ] Confirm Tier 4 sends urgent alert
|
|
- [ ] Verify no false positives from normal browsing
|
|
---
|
|
## Phase 3: Authelia on Staging
|
|
|
|
### Estimated time: 3-4 hours
|
|
|
|
### Step 3.1: Directory and secrets setup
|
|
- [ ] Create directory structure on staging: `/opt/docker/authelia/config/`
|
|
- [ ] Generate secrets: JWT, session, storage encryption, OIDC HMAC
|
|
- [ ] Generate password hashes for Travis and Jenny (argon2id)
|
|
- [ ] Create users_database.yml with both accounts
|
|
|
|
### Step 3.2: Authelia configuration
|
|
- [ ] Write configuration.yml:
|
|
- Session domain: staging domain
|
|
- Default policy: two_factor
|
|
- Storage: SQLite (local file)
|
|
- Notifier: filesystem (for staging — SMTP for production later)
|
|
- TOTP settings: SHA1, 6 digits, 30 second period
|
|
- [ ] Define access control rules:
|
|
- Portainer, Uptime Kuma, n8n, pbs-hub, phpMyAdmin, Traefik dashboard →
|
|
two_factor
|
|
- WordPress and public routes → bypass
|
|
|
|
### Step 3.3: Docker and Traefik integration
|
|
- [ ] Create Authelia docker-compose.yml
|
|
- [ ] Add Authelia to Traefik network
|
|
- [ ] Configure Traefik forwardAuth middleware pointing to Authelia
|
|
- [ ] Add middleware labels to all protected services
|
|
- [ ] Leave WordPress compose labels unchanged
|
|
|
|
### Step 3.4: Deploy and test
|
|
- [ ] Start Authelia container
|
|
- [ ] Navigate to a protected service — verify redirect to Authelia login
|
|
- [ ] Login with Travis account — verify redirect back to service
|
|
- [ ] Register TOTP device
|
|
- [ ] Verify 2FA prompt works
|
|
- [ ] Test SSO: login once, access other protected services without re-auth
|
|
- [ ] Verify WordPress login is unaffected
|
|
- [ ] Verify public site is unaffected
|
|
---
|
|
## Phase 4: Integration Testing on Staging
|
|
|
|
### Estimated time: 2 hours
|
|
|
|
### Step 4.1: Crowdsec + Authelia together
|
|
- [ ] Verify Crowdsec bouncer doesn't interfere with Authelia redirects
|
|
- [ ] Verify blocked IPs get Crowdsec block page, not Authelia login
|
|
- [ ] Test: brute force Authelia login → Crowdsec detects and blocks
|
|
- [ ] Verify request flow: Cloudflare → Traefik → Crowdsec → Authelia →
|
|
Service
|
|
|
|
### Step 4.2: Full stack validation
|
|
- [ ] All protected services accessible after Authelia login + 2FA
|
|
- [ ] WordPress admin login works independently
|
|
- [ ] Public site unaffected
|
|
- [ ] Crowdsec blocking bots silently
|
|
- [ ] Tier 3/4 alerts arriving in Google Chat
|
|
- [ ] No performance degradation
|
|
---
|
|
## Phase 5: Roll to Production
|
|
|
|
### Estimated time: 2-3 hours
|
|
|
|
### Step 5.1: Pre-production prep
|
|
- [ ] Document all staging config that worked
|
|
- [ ] Generate fresh secrets for production (never reuse staging secrets)
|
|
- [ ] Update Authelia session domain to production domain
|
|
- [ ] Switch Authelia notifier from filesystem to SMTP (Google Workspace)
|
|
- [ ] Update Crowdsec webhook URL to production n8n
|
|
|
|
### Step 5.2: Deploy Crowdsec to production
|
|
- [ ] Add Crowdsec to production docker-compose.yml
|
|
- [ ] Add Traefik bouncer plugin
|
|
- [ ] Add WordPress auth log source
|
|
- [ ] Enroll in Crowdsec console
|
|
- [ ] Verify blocking and alerting work
|
|
|
|
### Step 5.3: Deploy Authelia to production
|
|
- [ ] Add Authelia to production docker-compose.yml
|
|
- [ ] Add forwardAuth middleware to all protected services
|
|
- [ ] Verify SSO and 2FA work
|
|
- [ ] Register TOTP devices for Travis and Jenny
|
|
|
|
### Step 5.4: Post-deployment
|
|
- [ ] Monitor for 48 hours — watch for false positives
|
|
- [ ] Tune Crowdsec scenarios if needed
|
|
- [ ] Verify Wordfence lockout emails have dropped significantly
|
|
- [ ] Demote Wordfence to malware-scanning-only role
|
|
- [ ] Consider disabling Wordfence firewall features (now redundant)
|
|
---
|
|
## Phase 6: Codify in Ansible
|
|
|
|
### Estimated time: 3-4 hours
|
|
|
|
### Step 6.1: Crowdsec Ansible role
|
|
- [ ] Create Crowdsec tasks with Ansible tags (`--tags crowdsec`)
|
|
- [ ] Template docker-compose.yml with variables
|
|
- [ ] Template Crowdsec acquis.yaml (log sources)
|
|
- [ ] Store bouncer API key in ansible-vault
|
|
- [ ] Store console enrollment key in ansible-vault
|
|
|
|
### Step 6.2: Authelia Ansible role
|
|
- [ ] Create Authelia tasks with Ansible tags (`--tags authelia`)
|
|
- [ ] Template configuration.yml with variables
|
|
- [ ] Template users_database.yml
|
|
- [ ] Store all secrets in ansible-vault
|
|
- [ ] Add forwardAuth middleware labels to protected service templates
|
|
|
|
### Step 6.3: Validate
|
|
- [ ] Destroy staging and rebuild from Ansible
|
|
- [ ] Verify Crowdsec + Authelia come up correctly
|
|
- [ ] Verify alerting works after rebuild
|
|
- [ ] Verify SSO works after rebuild
|
|
---
|
|
## Wordfence Transition Plan
|
|
|
|
After Crowdsec is stable on production:
|
|
|
|
- [ ] Disable Wordfence firewall features (redundant with Crowdsec)
|
|
- [ ] Keep Wordfence for: scheduled malware scans (monthly), file integrity
|
|
checks
|
|
- [ ] Disable Wordfence login protection (Crowdsec handles this now)
|
|
- [ ] Disable lockout email notifications (the whole point!)
|
|
- [ ] Document remaining Wordfence role in tech wiki
|
|
- [ ] Long-term: when WordPress is replaced, Wordfence goes away entirely
|
|
---
|
|
## Security Maintenance Cadence
|
|
|
|
After full deployment:
|
|
|
|
### Daily (glance, 1 min)
|
|
- Check Uptime Kuma — green means good
|
|
- Respond to any Tier 3/4 Google Chat alerts
|
|
|
|
### Weekly (10 min)
|
|
- Review Crowdsec console dashboard for trends
|
|
- Check `cscli metrics` for volume patterns
|
|
- Update WordPress plugins on staging, then production
|
|
|
|
### Monthly (30 min)
|
|
- Run Wordfence malware scan
|
|
- Review WordPress user accounts
|
|
- Check Docker image updates (Traefik, WordPress, Crowdsec, Authelia)
|
|
- Review Cloudflare analytics
|
|
- Spot-check Crowdsec community blocklist enrollment
|
|
---
|
|
## Troubleshooting
|
|
|
|
### Crowdsec not blocking
|
|
bash
|
|
# Check if bouncer is connected
|
|
cscli bouncers list
|
|
|
|
# Check if decisions exist
|
|
cscli decisions list
|
|
|
|
# Check log parsing
|
|
cscli metrics
|
|
|
|
# Manually test a ban
|
|
cscli decisions add --ip 1.2.3.4 --reason "test ban" --duration 5m
|
|
|
|
|
|
### Authelia not redirecting
|
|
bash
|
|
# Check Authelia logs
|
|
docker logs authelia
|
|
|
|
# Verify Traefik middleware
|
|
docker exec traefik wget -qO- http://localhost:8080/api/http/middlewares
|
|
|
|
# Check forwardAuth is reaching Authelia
|
|
docker logs authelia | grep "authorization"
|
|
|
|
|
|
### False positives
|
|
bash
|
|
# Whitelist your own IP
|
|
cscli decisions delete --ip YOUR_IP
|
|
cscli parsers install crowdsecurity/whitelists
|
|
# Add your IP to /etc/crowdsec/parsers/s02-enrich/whitelists.yaml
|
|
|
|
|
|
### Locked out of Authelia
|
|
bash
|
|
# Reset user password
|
|
docker exec authelia authelia hash-password -- 'newpassword'
|
|
# Update users_database.yml with new hash
|
|
docker restart authelia
|
|
|
|
---
|
|
## Resources
|
|
|
|
### Crowdsec
|
|
- Official docs: https://docs.crowdsec.net/
|
|
- Traefik bouncer:
|
|
https://github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin
|
|
- WordPress collection:
|
|
https://hub.crowdsec.net/author/crowdsecurity/collections/wordpress
|
|
- Console: https://app.crowdsec.net/
|
|
|
|
### Authelia
|
|
- Official docs: https://www.authelia.com/
|
|
- Configuration reference:
|
|
https://www.authelia.com/configuration/prologue/introduction/
|
|
- Traefik integration: https://www.authelia.com/integration/proxies/traefik/
|
|
- File-based users: https://www.authelia.com/reference/guides/passwords/
|
|
|
|
### Traefik
|
|
- ForwardAuth middleware:
|
|
https://doc.traefik.io/traefik/middlewares/http/forwardauth/
|
|
- Plugin catalog: https://plugins.traefik.io/
|
|
---
|
|
## Success Criteria
|
|
|
|
- [ ] Wordfence lockout emails drop to near zero
|
|
- [ ] Crowdsec silently blocking bot traffic at Traefik layer
|
|
- [ ] Tier 3/4 alerts arriving in Google Chat when real threats detected
|
|
- [ ] All admin services behind Authelia SSO with 2FA
|
|
- [ ] WordPress auth unaffected
|
|
- [ ] Public site unaffected
|
|
- [ ] Everything codified in Ansible and reproducible
|
|
- [ ] Daily security overhead reduced from 15+ minutes to under 1 minute
|
|
---
|
|
*Last Updated: April 2, 2026*
|
|
*Maintained by: Travis*
|
|
*Project: Plant Based Southerner Infrastructure*
|
|
|
|
...sent from Jenny & Travis |