pbs-projects/Tech/Sessions/ssh-login-alerting.md

project	type	status	path	tags	created	updated
ssh-login-alerting	session-notes	active	Tech/Sessions	pbs security ssh auditd n8n monitoring	2026-04-21	2026-04-21

SSH Login Alerting — Session Notes (April 21, 2026)

What Got Done

• Installed Ansible on dev server (Ubuntu) via Ansible PPA
• Ran Lovebug's ssh_login_alert Ansible role against staging — first run was clean, no errors
• Confirmed auditd is running, audit rules are loaded, and the dispatcher plugin is wired to ssh-login-alert.sh
• Confirmed the n8n webhook URL works (manual curl triggers successfully)
• Confirmed USER_LOGIN events from sshd ARE being captured by auditd (ausearch -m USER_LOGIN returns them)
• Identified and fixed the exe= quote escaping mismatch (changed from escaped \" to single-quoted literal match)
• Confirmed the script receives events via stdin (debug logger proved it)

Current Blocker

The script matches USER_LOGIN events but fails silently at username extraction. The root cause:

USER_LOGIN events do NOT contain an acct= field. They use a different format:

type=USER_LOGIN msg=audit(...): pid=... uid=0 auid=1000 ses=387 subj=unconfined msg='op=login id=1000 exe="/usr/sbin/sshd" hostname=X addr=X terminal=/dev/pts/0 res=success'UID="root" AUID="tj57testadmin" ID="tj57testadmin"

Key differences from what the script expects:

• Username is in AUID="username" (enriched suffix), not acct="username"
• Also available as ID="username" in the enriched suffix
• Numeric auid=1000 is in the main body but needs getent passwd to resolve
• id=1000 inside the msg='...' block is also numeric

The script hits [[ -n "$username" ]] || continue with an empty username and silently skips the event.

An AUID regex fallback was added but still failed — needs further debugging.

Next Steps — Debugging Checklist

• SSH into staging and capture raw event: sudo ausearch -m USER_LOGIN --start recent --raw | tail -1

• Test regex patterns manually against the raw event line in bash to confirm which pattern actually matches

• Key test patterns to try in bash: bash line=''

  Test 1: AUID with quotes

  && echo "AUID match: ${BASH_REMATCH[1]}" || echo "no AUID match"

  Test 2: acct with quotes

  && echo "acct match: ${BASH_REMATCH[1]}" || echo "no acct match"

  Test 3: ID with quotes

  && echo "ID match: ${BASH_REMATCH[1]}" || echo "no ID match"

  Test 4: auid numeric then resolve

  && echo "auid match: ${BASH_REMATCH[1]}" || echo "no auid match"

• Once the working regex is identified, update the username extraction block in /usr/local/bin/ssh-login-alert.sh

• Restart auditd: sudo service auditd restart

• SSH in from another terminal and check: sudo journalctl -t ssh-login-alert --since "1 minute ago" --no-pager -o cat

• Confirm Google Chat notification arrives

• Once working on staging, backport the fix into the Ansible role template

• Remove the old commented-out exe= line from the script

Architecture Reminder

SSH login → auditd captures USER_LOGIN event → audispd plugin pipes to ssh-login-alert.sh → script extracts fields + POSTs JSON to n8n webhook → n8n formats + sends to Google Chat

Useful Debug Commands

bash

Check auditd status

sudo systemctl status auditd

Check loaded audit rules

sudo auditctl -l

Search for SSH login events

sudo ausearch -m USER_LOGIN --start recent

Raw event format (what the script actually sees)

sudo ausearch -m USER_LOGIN --start recent --raw | tail -1

Tail audit log for live events

sudo tail -f /var/log/audit/audit.log

Check script logs

sudo journalctl -t ssh-login-alert --since "5 minutes ago" --no-pager -o cat

Restart auditd (picks up script changes)

sudo service auditd restart

Verify config file

sudo cat /etc/pbs-alerts/config

Learnings

• USER_LOGIN is a PAM userspace event, not a syscall — audit rule key= tags don't apply to it
• The exe= field in USER_LOGIN events uses literal double quotes, not escaped quotes — use single-quoted bash patterns for matching
• auditd dispatcher plugins (type = always) receive ALL events — filtering must happen in the script
• Debug logger lines spawning per-event can overwhelm auditd's plugin queue — use sparingly and remove after testing
• sudo ausearch -m USER_LOGIN --start recent --raw is the best way to see exactly what the script receives

Open Items from Session

• Ansible user strategy: pbsdeploy (no password, no sudo, key-only) is for CI/CD only. Ansible playbooks should run as admin account with -K for sudo. These are separate concerns — revisit when CI/CD pipeline discussion comes back up.

---

Session Date: April 21, 2026 Next Session: Continue debugging username regex, get end-to-end alert working

...sent from Jenny & Travis