Create security-incident-response.md via n8n

PBS/Tech/Sessions/security-incident-response.md/security-incident-response.md

@@ -0,0 +1,96 @@
+---
+project: security-incident-response
+type: Tech/Sessions
+status: complete
+tags: security, wordpress, apache, cloudflare, wordfence, docker
+created: 2026-04-07
+updated: 2026-04-07
+path: PBS/Tech/Sessions/security-incident-response.md
+---
+
+# Security Incident Response Session
+
+## What Happened
+
+WordPress was experiencing repeated OOM (out of memory) crashes during
+the evening. The site was going down intermittently while Jenny was
+saving posts. Investigation revealed a sustained multi-day brute force
+attack on /wp-login.php that had gone undetected, combined with
+simultaneous bot crawling and Jenny's post saves creating a perfect
+storm of Apache worker exhaustion.
+
+## Root Cause
+
+WordPress container (limited to 1.953GiB) was hitting its memory
+ceiling repeatedly. Each Apache mpm_prefork worker loads the full
+WordPress + plugin stack independently (~200MiB per active worker).
+With brute force bots hammering wp-login.php (11,000+ hits over
+several days), Googlebot and Facebook crawling simultaneously, and
+Elementor saving posts, Apache spawned too many workers and exhausted
+available RAM. Swap filled to 479MiB/496MiB.
+
+## Key Findings
+
+- 11,022 wp-login.php hits found in WordPress Docker logs over several days
+- Wordfence rate limiting was set to Unlimited — logging attacks but
+blocking nothing
+- Cloudflare WAF had no rule protecting wp-login.php
+- WordPress plugin stack was bloated with 7+ unused/redundant plugins
+loading on every request
+- Apache mpm_prefork has no worker cap by default — spawns unlimited
+workers under load
+- Swap does not auto-clear after OOM events — requires manual swapoff/swapon
+
+## Fixes Applied
+
+### Immediate
+- Restarted WordPress container to clear zombie Apache workers
+- Blocked attacking IP in Wordfence manually
+- Flushed swap: sudo swapoff -a && sudo swapon -a
+
+### Cloudflare
+- Added Managed Challenge rule on /wp-login.php — challenges all
+requests before reaching server
+- Existing rate limit rule (5 requests/10 seconds) confirmed working —
+35 hits in 24 hours
+- Confirmed brute force dropped from hundreds/hour to 6/hour after
+rule activation
+
+### Wordfence
+- Enabled rate limiting (was set to Unlimited)
+- Set anyone's requests to 240/min → block
+- Set crawler page views to 60/min → block
+- Set 404s to 10/min → block for both humans and crawlers
+- Set login lockout to 3 failed attempts
+- Block duration: 1 hour
+
+### Plugin Cleanup (7 removed)
+- Hello Dolly (joke plugin)
+- Transients Manager (dev tool)
+- WPForms Lite (redundant — had WPZOOM Forms)
+- WPZOOM Forms (neither form plugin was in use)
+- WooCommerce Shipping (not in use yet)
+- Templately (unused template library)
+- WP Crontrol (dev tool — wpcron sidecar handles this)
+
+## Current Protection Stack
+
+- Cloudflare Managed Challenge → stops bots on first request
+- Cloudflare rate limit → 5 requests/10 seconds → block
+- Wordfence rate limiting → enabled with sensible limits
+- Wordfence login lockout → 3 attempts → 1 hour block
+
+## Monitoring
+
+- Cloudflare Security Events → shows challenges/blocks before hitting server
+- Wordfence Login attempts → shows what gets through Cloudflare
+- Docker logs ground truth: docker logs wordpress 2>&1 | grep
+"wp-login.php" | wc -l
+
+## Next Steps
+
+- Build WordPress log monitoring pipeline (see separate project note)
+- Tune Apache mpm_prefork worker limits (now understood, not urgent)
+- Evaluate PHP-FPM as longer term replacement for Apache prefork
+- Consider removing/replacing Jetpack with lightweight alternatives
+- Continue plugin audit — Ultimate Member, Google Site Kit still candidates