Create ssh-login-alerting.md via n8n

Tech/Sessions/ssh-login-alerting.md

@@ -0,0 +1,153 @@
+---
+project: ssh-login-alerting
+type: session-notes
+status: active
+path: Tech/Sessions
+tags:
+  - pbs
+  - security
+  - ssh
+  - auditd
+  - n8n
+  - monitoring
+created: 2026-04-21
+updated: 2026-04-21
+---
+# SSH Login Alerting — Session Notes (April 21, 2026)
+
+## What Got Done
+
+- Installed Ansible on dev server (Ubuntu) via Ansible PPA
+- Ran Lovebug's `ssh_login_alert` Ansible role against staging — first run
+was clean, no errors
+- Confirmed auditd is running, audit rules are loaded, and the dispatcher
+plugin is wired to `ssh-login-alert.sh`
+- Confirmed the n8n webhook URL works (manual curl triggers successfully)
+- Confirmed `USER_LOGIN` events from sshd ARE being captured by auditd
+(`ausearch -m USER_LOGIN` returns them)
+- Identified and fixed the exe= quote escaping mismatch (changed from
+escaped `\"` to single-quoted literal match)
+- Confirmed the script receives events via stdin (debug logger proved it)
+
+## Current Blocker
+
+The script matches `USER_LOGIN` events but fails silently at username
+extraction. The root cause:
+
+`USER_LOGIN` events do NOT contain an `acct=` field. They use a different
+format:
+
+
+type=USER_LOGIN msg=audit(...): pid=... uid=0 auid=1000 ses=387
+subj=unconfined msg='op=login id=1000 exe="/usr/sbin/sshd" hostname=X
+addr=X terminal=/dev/pts/0 res=success'UID="root" AUID="tj57testadmin"
+ID="tj57testadmin"
+
+
+Key differences from what the script expects:
+- Username is in `AUID="username"` (enriched suffix), not `acct="username"`
+- Also available as `ID="username"` in the enriched suffix
+- Numeric `auid=1000` is in the main body but needs `getent passwd` to
+resolve
+- `id=1000` inside the `msg='...'` block is also numeric
+
+The script hits `[[ -n "$username" ]] || continue` with an empty username
+and silently skips the event.
+
+An `AUID` regex fallback was added but still failed — needs further
+debugging.
+
+## Next Steps — Debugging Checklist
+
+- [ ] SSH into staging and capture raw event: `sudo ausearch -m USER_LOGIN
+--start recent --raw | tail -1`
+- [ ] Test regex patterns manually against the raw event line in bash to
+confirm which pattern actually matches
+- [ ] Key test patterns to try in bash:
+  bash
+  line=''
+  # Test 1: AUID with quotes
+  [[ "$line" =~ AUID=\"([^\"]+)\" ]] && echo "AUID match:
+${BASH_REMATCH[1]}" || echo "no AUID match"
+  # Test 2: acct with quotes
+  [[ "$line" =~ acct=\"([^\"]+)\" ]] && echo "acct match:
+${BASH_REMATCH[1]}" || echo "no acct match"
+  # Test 3: ID with quotes
+  [[ "$line" =~ ID=\"([^\"]+)\" ]] && echo "ID match: ${BASH_REMATCH[1]}"
+|| echo "no ID match"
+  # Test 4: auid numeric then resolve
+  [[ "$line" =~ auid=([0-9]+) ]] && echo "auid match: ${BASH_REMATCH[1]}"
+|| echo "no auid match"
+  
+- [ ] Once the working regex is identified, update the username extraction
+block in `/usr/local/bin/ssh-login-alert.sh`
+- [ ] Restart auditd: `sudo service auditd restart`
+- [ ] SSH in from another terminal and check: `sudo journalctl -t
+ssh-login-alert --since "1 minute ago" --no-pager -o cat`
+- [ ] Confirm Google Chat notification arrives
+- [ ] Once working on staging, backport the fix into the Ansible role
+template
+- [ ] Remove the old commented-out exe= line from the script
+
+## Architecture Reminder
+
+
+SSH login → auditd captures USER_LOGIN event → audispd plugin pipes to
+ssh-login-alert.sh →
+script extracts fields + POSTs JSON to n8n webhook → n8n formats + sends to
+Google Chat
+
+
+## Useful Debug Commands
+
+bash
+# Check auditd status
+sudo systemctl status auditd
+
+# Check loaded audit rules
+sudo auditctl -l
+
+# Search for SSH login events
+sudo ausearch -m USER_LOGIN --start recent
+
+# Raw event format (what the script actually sees)
+sudo ausearch -m USER_LOGIN --start recent --raw | tail -1
+
+# Tail audit log for live events
+sudo tail -f /var/log/audit/audit.log
+
+# Check script logs
+sudo journalctl -t ssh-login-alert --since "5 minutes ago" --no-pager -o cat
+
+# Restart auditd (picks up script changes)
+sudo service auditd restart
+
+# Verify config file
+sudo cat /etc/pbs-alerts/config
+
+
+## Learnings
+
+- `USER_LOGIN` is a PAM userspace event, not a syscall — audit rule `key=`
+tags don't apply to it
+- The exe= field in USER_LOGIN events uses literal double quotes, not
+escaped quotes — use single-quoted bash patterns for matching
+- auditd dispatcher plugins (`type = always`) receive ALL events —
+filtering must happen in the script
+- Debug logger lines spawning per-event can overwhelm auditd's plugin queue
+— use sparingly and remove after testing
+- `sudo ausearch -m USER_LOGIN --start recent --raw` is the best way to see
+exactly what the script receives
+
+## Open Items from Session
+
+- Ansible user strategy: `pbsdeploy` (no password, no sudo, key-only) is
+for CI/CD only. Ansible playbooks should run as admin account with `-K` for
+sudo. These are separate concerns — revisit when CI/CD pipeline discussion
+comes back up.
+---
+*Session Date: April 21, 2026*
+*Next Session: Continue debugging username regex, get end-to-end alert
+working*
+
+...sent from Jenny & Travis