Update ssh-login-alerting.md via n8n

Tech/Projects/ssh-login-alerting.md

@@ -7,13 +7,13 @@ tags:
   - pbs
   - security
   - ssh
-  - auditd
+  - journald
+  - python
   - n8n
   - monitoring
 created: 2026-04-19
-updated: 2026-04-19
+updated: 2026-04-21
 ---
-
 # SSH Login Alerting
 
 ## Goal
@@ -25,24 +25,39 @@ key-based account.
 
 ## Architecture
 
-```
-SSH login → auditd captures event → trigger script parses it →
-curl POST to n8n webhook → n8n formats + sends to Google Chat
-```
+
+sshd → journald → Python sdjournal tailer → POST to n8n webhook →
+n8n formats + sends to Google Chat
+
 
 ## Design Decisions
 
-- **auditd over log-parsing** — kernel-level event capture, more reliable
-than tailing `/var/log/auth.log`.
-- **Shell script as middleman** — dumb, single-purpose: parse auditd event,
-POST to n8n webhook. No logic in the script beyond formatting.
-- **n8n handles notification routing** — keeps all alert
-formatting/delivery centralized where other PBS notifications already live.
-Server just fires events; n8n decides what to do with them.
-- **Alert on every login for v1** — simpler than trying to define
-"unexpected" up front. Tune later if noise becomes a problem.
-- **Scope: all accounts, all PBS servers** — staging and production, every
-user (`pbsdeploy`, admin, root, etc.).
+- **journald over auditd** — v1 attempted `auditd → shell dispatcher →
+regex → webhook` and was abandoned after silent event drops and two hours
+of failed debugging. Root cause: auditd emits multiple records per logical
+event with format variations that break line-by-line regex parsing.
+Journald is one-line-per-event with structured metadata
+(`_SYSTEMD_UNIT=ssh.service`), which eliminates the multi-record coalescing
+problem entirely.
+- **Python sdjournal tailer, not pam_exec** — pam_exec would give fully
+structured data with zero regex via PAM env vars (`PAM_USER`, `PAM_RHOST`,
+etc.), but touches the SSH auth path. Explicit tradeoff: accept one
+remaining regex pattern in exchange for zero risk to login infrastructure.
+If the tailer crashes, SSH keeps working.
+- **Cursor persistence solves the silent-drop problem** —
+`systemd.journal.Reader.seek_cursor()` with cursor saved to disk after each
+successful webhook POST gives exactly-once delivery across restarts,
+crashes, and n8n outages. This was the actual failure mode of v1.
+- **Regex is contained and testable** — one pattern against sshd's
+`MESSAGE` field (stable across OpenSSH versions for ~10 years), in one
+Python function, with unit tests against fixture strings. When it breaks,
+fix is one line in one place.
+- **n8n handles notification routing** — unchanged from v1. Server fires
+events; n8n formats and delivers.
+- **Alert on every login for v1** — unchanged. Tune later if noise becomes
+a problem.
+- **Scope: all accounts, all PBS servers** — unchanged. Staging and
+production, every user.
 
 ## Why This Matters
 
@@ -54,26 +69,42 @@ main threat vector.
 
 ## Next Steps
 
-- [ ] Configure auditd rules on staging server to capture SSH login events
-- [ ] Write shell script that parses auditd events and POSTs to n8n webhook
+- [ ] Decide deployment model: host systemd service vs containerized
+(leaning host systemd — simpler, more reliable for journal access)
+- [ ] Write Python tailer using `systemd.journal.Reader` with
+`add_match(_SYSTEMD_UNIT="ssh.service")`, cursor persistence, and HTTP POST
+to n8n
+- [ ] Unit test the MESSAGE regex against fixture strings (accepted
+publickey, accepted password, failed attempts, invalid user)
+- [ ] Deploy to staging, test cursor behavior across restarts
 - [ ] Build n8n workflow: webhook trigger → format message → send to Google
 Chat
-- [ ] Test end-to-end on staging (SSH in, confirm alert arrives)
-- [ ] Roll out to production
-- [ ] Decide on Chat space (webadmin? dedicated security space?)
+- [ ] End-to-end test on staging (SSH in, kill tailer, SSH in again,
+restart tailer, confirm replay)
+- [ ] Add Uptime Kuma monitor for the tailer service
+- [ ] Roll out to production via Ansible
+- [ ] Decommission v1 auditd dispatcher
 
 ## Open Questions
 
-- Which Google Chat space should alerts go to?
-- Do we want any metadata in the alert beyond username + source IP +
-timestamp (e.g., geolocation lookup on source IP)?
-- Do we want a separate "critical" alert path for root logins?
+- Host systemd service or containerized? Leaning host service — touches
+host-level journal, simpler bind-mount story. Breaks Compose aesthetic but
+fits the job.
+- Where does cursor state live? (Path, ownership, permissions — probably
+`/var/lib/ssh-login-alerter/cursor`.)
+- Which Google Chat space — `webadmin` or a dedicated security space?
+- Metadata beyond username + source IP + timestamp (e.g., geolocation on
+source IP)?
+- Separate "critical" alert path for root logins?
+- How does Uptime Kuma know the tailer is alive? Heartbeat ping from the
+tailer itself, or process check?
 
 ## Priority
 
-Higher priority than env file hardening — it's a force multiplier. Even if
-the env hardening isn't done yet, knowing about an unauthorized login
-immediately limits blast radius.
+Still higher priority than env file hardening. v1's silent drops mean we
+currently have *no* working alert on the `pbsdeploy` account — the
+force-multiplier argument is sharper now than it was at project inception.
 
+...sent from Jenny & Travis
 
 ...sent from Jenny & Travis