diff --git a/Tech/Sessions/ssh-login-alerting.md b/Tech/Sessions/ssh-login-alerting.md new file mode 100644 index 0000000..279a265 --- /dev/null +++ b/Tech/Sessions/ssh-login-alerting.md @@ -0,0 +1,153 @@ +--- +project: ssh-login-alerting +type: session-notes +status: active +path: Tech/Sessions +tags: + - pbs + - security + - ssh + - auditd + - n8n + - monitoring +created: 2026-04-21 +updated: 2026-04-21 +--- +# SSH Login Alerting — Session Notes (April 21, 2026) + +## What Got Done + +- Installed Ansible on dev server (Ubuntu) via Ansible PPA +- Ran Lovebug's `ssh_login_alert` Ansible role against staging — first run +was clean, no errors +- Confirmed auditd is running, audit rules are loaded, and the dispatcher +plugin is wired to `ssh-login-alert.sh` +- Confirmed the n8n webhook URL works (manual curl triggers successfully) +- Confirmed `USER_LOGIN` events from sshd ARE being captured by auditd +(`ausearch -m USER_LOGIN` returns them) +- Identified and fixed the exe= quote escaping mismatch (changed from +escaped `\"` to single-quoted literal match) +- Confirmed the script receives events via stdin (debug logger proved it) + +## Current Blocker + +The script matches `USER_LOGIN` events but fails silently at username +extraction. The root cause: + +`USER_LOGIN` events do NOT contain an `acct=` field. They use a different +format: + + +type=USER_LOGIN msg=audit(...): pid=... uid=0 auid=1000 ses=387 +subj=unconfined msg='op=login id=1000 exe="/usr/sbin/sshd" hostname=X +addr=X terminal=/dev/pts/0 res=success'UID="root" AUID="tj57testadmin" +ID="tj57testadmin" + + +Key differences from what the script expects: +- Username is in `AUID="username"` (enriched suffix), not `acct="username"` +- Also available as `ID="username"` in the enriched suffix +- Numeric `auid=1000` is in the main body but needs `getent passwd` to +resolve +- `id=1000` inside the `msg='...'` block is also numeric + +The script hits `[[ -n "$username" ]] || continue` with an empty username +and silently skips the event. + +An `AUID` regex fallback was added but still failed — needs further +debugging. + +## Next Steps — Debugging Checklist + +- [ ] SSH into staging and capture raw event: `sudo ausearch -m USER_LOGIN +--start recent --raw | tail -1` +- [ ] Test regex patterns manually against the raw event line in bash to +confirm which pattern actually matches +- [ ] Key test patterns to try in bash: + bash + line='' + # Test 1: AUID with quotes + [[ "$line" =~ AUID=\"([^\"]+)\" ]] && echo "AUID match: +${BASH_REMATCH[1]}" || echo "no AUID match" + # Test 2: acct with quotes + [[ "$line" =~ acct=\"([^\"]+)\" ]] && echo "acct match: +${BASH_REMATCH[1]}" || echo "no acct match" + # Test 3: ID with quotes + [[ "$line" =~ ID=\"([^\"]+)\" ]] && echo "ID match: ${BASH_REMATCH[1]}" +|| echo "no ID match" + # Test 4: auid numeric then resolve + [[ "$line" =~ auid=([0-9]+) ]] && echo "auid match: ${BASH_REMATCH[1]}" +|| echo "no auid match" + +- [ ] Once the working regex is identified, update the username extraction +block in `/usr/local/bin/ssh-login-alert.sh` +- [ ] Restart auditd: `sudo service auditd restart` +- [ ] SSH in from another terminal and check: `sudo journalctl -t +ssh-login-alert --since "1 minute ago" --no-pager -o cat` +- [ ] Confirm Google Chat notification arrives +- [ ] Once working on staging, backport the fix into the Ansible role +template +- [ ] Remove the old commented-out exe= line from the script + +## Architecture Reminder + + +SSH login → auditd captures USER_LOGIN event → audispd plugin pipes to +ssh-login-alert.sh → +script extracts fields + POSTs JSON to n8n webhook → n8n formats + sends to +Google Chat + + +## Useful Debug Commands + +bash +# Check auditd status +sudo systemctl status auditd + +# Check loaded audit rules +sudo auditctl -l + +# Search for SSH login events +sudo ausearch -m USER_LOGIN --start recent + +# Raw event format (what the script actually sees) +sudo ausearch -m USER_LOGIN --start recent --raw | tail -1 + +# Tail audit log for live events +sudo tail -f /var/log/audit/audit.log + +# Check script logs +sudo journalctl -t ssh-login-alert --since "5 minutes ago" --no-pager -o cat + +# Restart auditd (picks up script changes) +sudo service auditd restart + +# Verify config file +sudo cat /etc/pbs-alerts/config + + +## Learnings + +- `USER_LOGIN` is a PAM userspace event, not a syscall — audit rule `key=` +tags don't apply to it +- The exe= field in USER_LOGIN events uses literal double quotes, not +escaped quotes — use single-quoted bash patterns for matching +- auditd dispatcher plugins (`type = always`) receive ALL events — +filtering must happen in the script +- Debug logger lines spawning per-event can overwhelm auditd's plugin queue +— use sparingly and remove after testing +- `sudo ausearch -m USER_LOGIN --start recent --raw` is the best way to see +exactly what the script receives + +## Open Items from Session + +- Ansible user strategy: `pbsdeploy` (no password, no sudo, key-only) is +for CI/CD only. Ansible playbooks should run as admin account with `-K` for +sudo. These are separate concerns — revisit when CI/CD pipeline discussion +comes back up. +--- +*Session Date: April 21, 2026* +*Next Session: Continue debugging username regex, get end-to-end alert +working* + +...sent from Jenny & Travis \ No newline at end of file