3.5 KiB
| project | type | status | tags | created | updated | path |
|---|---|---|---|---|---|---|
| security-incident-response | Tech/Sessions | complete | security, wordpress, apache, cloudflare, wordfence, docker | 2026-04-07 | 2026-04-07 | PBS/Tech/Sessions/security-incident-response.md |
Security Incident Response Session
What Happened
WordPress was experiencing repeated OOM (out of memory) crashes during the evening. The site was going down intermittently while Jenny was saving posts. Investigation revealed a sustained multi-day brute force attack on /wp-login.php that had gone undetected, combined with simultaneous bot crawling and Jenny's post saves creating a perfect storm of Apache worker exhaustion.
Root Cause
WordPress container (limited to 1.953GiB) was hitting its memory ceiling repeatedly. Each Apache mpm_prefork worker loads the full WordPress + plugin stack independently (~200MiB per active worker). With brute force bots hammering wp-login.php (11,000+ hits over several days), Googlebot and Facebook crawling simultaneously, and Elementor saving posts, Apache spawned too many workers and exhausted available RAM. Swap filled to 479MiB/496MiB.
Key Findings
- 11,022 wp-login.php hits found in WordPress Docker logs over several days
- Wordfence rate limiting was set to Unlimited — logging attacks but blocking nothing
- Cloudflare WAF had no rule protecting wp-login.php
- WordPress plugin stack was bloated with 7+ unused/redundant plugins loading on every request
- Apache mpm_prefork has no worker cap by default — spawns unlimited workers under load
- Swap does not auto-clear after OOM events — requires manual swapoff/swapon
Fixes Applied
Immediate
- Restarted WordPress container to clear zombie Apache workers
- Blocked attacking IP in Wordfence manually
- Flushed swap: sudo swapoff -a && sudo swapon -a
Cloudflare
- Added Managed Challenge rule on /wp-login.php — challenges all requests before reaching server
- Existing rate limit rule (5 requests/10 seconds) confirmed working — 35 hits in 24 hours
- Confirmed brute force dropped from hundreds/hour to 6/hour after rule activation
Wordfence
- Enabled rate limiting (was set to Unlimited)
- Set anyone's requests to 240/min → block
- Set crawler page views to 60/min → block
- Set 404s to 10/min → block for both humans and crawlers
- Set login lockout to 3 failed attempts
- Block duration: 1 hour
Plugin Cleanup (7 removed)
- Hello Dolly (joke plugin)
- Transients Manager (dev tool)
- WPForms Lite (redundant — had WPZOOM Forms)
- WPZOOM Forms (neither form plugin was in use)
- WooCommerce Shipping (not in use yet)
- Templately (unused template library)
- WP Crontrol (dev tool — wpcron sidecar handles this)
Current Protection Stack
- Cloudflare Managed Challenge → stops bots on first request
- Cloudflare rate limit → 5 requests/10 seconds → block
- Wordfence rate limiting → enabled with sensible limits
- Wordfence login lockout → 3 attempts → 1 hour block
Monitoring
- Cloudflare Security Events → shows challenges/blocks before hitting server
- Wordfence Login attempts → shows what gets through Cloudflare
- Docker logs ground truth: docker logs wordpress 2>&1 | grep "wp-login.php" | wc -l
Next Steps
- Build WordPress log monitoring pipeline (see separate project note)
- Tune Apache mpm_prefork worker limits (now understood, not urgent)
- Evaluate PHP-FPM as longer term replacement for Apache prefork
- Consider removing/replacing Jetpack with lightweight alternatives
- Continue plugin audit — Ultimate Member, Google Site Kit still candidates