---
project: ssh-login-alerting
type: session-notes
status: active
path: Tech/Sessions
tags:
  - pbs
  - security
  - ssh
  - auditd
  - n8n
  - monitoring
created: 2026-04-21
updated: 2026-04-21
---
# SSH Login Alerting — Session Notes (April 21, 2026)

## What Got Done

- Installed Ansible on dev server (Ubuntu) via Ansible PPA
- Ran Lovebug's `ssh_login_alert` Ansible role against staging — first run
was clean, no errors
- Confirmed auditd is running, audit rules are loaded, and the dispatcher
plugin is wired to `ssh-login-alert.sh`
- Confirmed the n8n webhook URL works (manual curl triggers successfully)
- Confirmed `USER_LOGIN` events from sshd ARE being captured by auditd
(`ausearch -m USER_LOGIN` returns them)
- Identified and fixed the exe= quote escaping mismatch (changed from
escaped `\"` to single-quoted literal match)
- Confirmed the script receives events via stdin (debug logger proved it)

## Current Blocker

The script matches `USER_LOGIN` events but fails silently at username
extraction. The root cause:

`USER_LOGIN` events do NOT contain an `acct=` field. They use a different
format:


type=USER_LOGIN msg=audit(...): pid=... uid=0 auid=1000 ses=387
subj=unconfined msg='op=login id=1000 exe="/usr/sbin/sshd" hostname=X
addr=X terminal=/dev/pts/0 res=success'UID="root" AUID="tj57testadmin"
ID="tj57testadmin"


Key differences from what the script expects:
- Username is in `AUID="username"` (enriched suffix), not `acct="username"`
- Also available as `ID="username"` in the enriched suffix
- Numeric `auid=1000` is in the main body but needs `getent passwd` to
resolve
- `id=1000` inside the `msg='...'` block is also numeric

The script hits `[[ -n "$username" ]] || continue` with an empty username
and silently skips the event.

An `AUID` regex fallback was added but still failed — needs further
debugging.

## Next Steps — Debugging Checklist

- [ ] SSH into staging and capture raw event: `sudo ausearch -m USER_LOGIN
--start recent --raw | tail -1`
- [ ] Test regex patterns manually against the raw event line in bash to
confirm which pattern actually matches
- [ ] Key test patterns to try in bash:
  bash
  line=''
  # Test 1: AUID with quotes
  [[ "$line" =~ AUID=\"([^\"]+)\" ]] && echo "AUID match:
${BASH_REMATCH[1]}" || echo "no AUID match"
  # Test 2: acct with quotes
  [[ "$line" =~ acct=\"([^\"]+)\" ]] && echo "acct match:
${BASH_REMATCH[1]}" || echo "no acct match"
  # Test 3: ID with quotes
  [[ "$line" =~ ID=\"([^\"]+)\" ]] && echo "ID match: ${BASH_REMATCH[1]}"
|| echo "no ID match"
  # Test 4: auid numeric then resolve
  [[ "$line" =~ auid=([0-9]+) ]] && echo "auid match: ${BASH_REMATCH[1]}"
|| echo "no auid match"
  
- [ ] Once the working regex is identified, update the username extraction
block in `/usr/local/bin/ssh-login-alert.sh`
- [ ] Restart auditd: `sudo service auditd restart`
- [ ] SSH in from another terminal and check: `sudo journalctl -t
ssh-login-alert --since "1 minute ago" --no-pager -o cat`
- [ ] Confirm Google Chat notification arrives
- [ ] Once working on staging, backport the fix into the Ansible role
template
- [ ] Remove the old commented-out exe= line from the script

## Architecture Reminder


SSH login → auditd captures USER_LOGIN event → audispd plugin pipes to
ssh-login-alert.sh →
script extracts fields + POSTs JSON to n8n webhook → n8n formats + sends to
Google Chat


## Useful Debug Commands

bash
# Check auditd status
sudo systemctl status auditd

# Check loaded audit rules
sudo auditctl -l

# Search for SSH login events
sudo ausearch -m USER_LOGIN --start recent

# Raw event format (what the script actually sees)
sudo ausearch -m USER_LOGIN --start recent --raw | tail -1

# Tail audit log for live events
sudo tail -f /var/log/audit/audit.log

# Check script logs
sudo journalctl -t ssh-login-alert --since "5 minutes ago" --no-pager -o cat

# Restart auditd (picks up script changes)
sudo service auditd restart

# Verify config file
sudo cat /etc/pbs-alerts/config


## Learnings

- `USER_LOGIN` is a PAM userspace event, not a syscall — audit rule `key=`
tags don't apply to it
- The exe= field in USER_LOGIN events uses literal double quotes, not
escaped quotes — use single-quoted bash patterns for matching
- auditd dispatcher plugins (`type = always`) receive ALL events —
filtering must happen in the script
- Debug logger lines spawning per-event can overwhelm auditd's plugin queue
— use sparingly and remove after testing
- `sudo ausearch -m USER_LOGIN --start recent --raw` is the best way to see
exactly what the script receives

## Open Items from Session

- Ansible user strategy: `pbsdeploy` (no password, no sudo, key-only) is
for CI/CD only. Ansible playbooks should run as admin account with `-K` for
sudo. These are separate concerns — revisit when CI/CD pipeline discussion
comes back up.
---
*Session Date: April 21, 2026*
*Next Session: Continue debugging username regex, get end-to-end alert
working*

...sent from Jenny & Travis