diff --git a/Tech/Projects/ssh-login-alerting.md b/Tech/Projects/ssh-login-alerting.md index 40f91ab..85e91cc 100644 --- a/Tech/Projects/ssh-login-alerting.md +++ b/Tech/Projects/ssh-login-alerting.md @@ -7,13 +7,13 @@ tags: - pbs - security - ssh - - auditd + - journald + - python - n8n - monitoring created: 2026-04-19 -updated: 2026-04-19 +updated: 2026-04-21 --- - # SSH Login Alerting ## Goal @@ -25,24 +25,39 @@ key-based account. ## Architecture -``` -SSH login → auditd captures event → trigger script parses it → -curl POST to n8n webhook → n8n formats + sends to Google Chat -``` + +sshd → journald → Python sdjournal tailer → POST to n8n webhook → +n8n formats + sends to Google Chat + ## Design Decisions -- **auditd over log-parsing** — kernel-level event capture, more reliable -than tailing `/var/log/auth.log`. -- **Shell script as middleman** — dumb, single-purpose: parse auditd event, -POST to n8n webhook. No logic in the script beyond formatting. -- **n8n handles notification routing** — keeps all alert -formatting/delivery centralized where other PBS notifications already live. -Server just fires events; n8n decides what to do with them. -- **Alert on every login for v1** — simpler than trying to define -"unexpected" up front. Tune later if noise becomes a problem. -- **Scope: all accounts, all PBS servers** — staging and production, every -user (`pbsdeploy`, admin, root, etc.). +- **journald over auditd** — v1 attempted `auditd → shell dispatcher → +regex → webhook` and was abandoned after silent event drops and two hours +of failed debugging. Root cause: auditd emits multiple records per logical +event with format variations that break line-by-line regex parsing. +Journald is one-line-per-event with structured metadata +(`_SYSTEMD_UNIT=ssh.service`), which eliminates the multi-record coalescing +problem entirely. +- **Python sdjournal tailer, not pam_exec** — pam_exec would give fully +structured data with zero regex via PAM env vars (`PAM_USER`, `PAM_RHOST`, +etc.), but touches the SSH auth path. Explicit tradeoff: accept one +remaining regex pattern in exchange for zero risk to login infrastructure. +If the tailer crashes, SSH keeps working. +- **Cursor persistence solves the silent-drop problem** — +`systemd.journal.Reader.seek_cursor()` with cursor saved to disk after each +successful webhook POST gives exactly-once delivery across restarts, +crashes, and n8n outages. This was the actual failure mode of v1. +- **Regex is contained and testable** — one pattern against sshd's +`MESSAGE` field (stable across OpenSSH versions for ~10 years), in one +Python function, with unit tests against fixture strings. When it breaks, +fix is one line in one place. +- **n8n handles notification routing** — unchanged from v1. Server fires +events; n8n formats and delivers. +- **Alert on every login for v1** — unchanged. Tune later if noise becomes +a problem. +- **Scope: all accounts, all PBS servers** — unchanged. Staging and +production, every user. ## Why This Matters @@ -54,26 +69,42 @@ main threat vector. ## Next Steps -- [ ] Configure auditd rules on staging server to capture SSH login events -- [ ] Write shell script that parses auditd events and POSTs to n8n webhook +- [ ] Decide deployment model: host systemd service vs containerized +(leaning host systemd — simpler, more reliable for journal access) +- [ ] Write Python tailer using `systemd.journal.Reader` with +`add_match(_SYSTEMD_UNIT="ssh.service")`, cursor persistence, and HTTP POST +to n8n +- [ ] Unit test the MESSAGE regex against fixture strings (accepted +publickey, accepted password, failed attempts, invalid user) +- [ ] Deploy to staging, test cursor behavior across restarts - [ ] Build n8n workflow: webhook trigger → format message → send to Google Chat -- [ ] Test end-to-end on staging (SSH in, confirm alert arrives) -- [ ] Roll out to production -- [ ] Decide on Chat space (webadmin? dedicated security space?) +- [ ] End-to-end test on staging (SSH in, kill tailer, SSH in again, +restart tailer, confirm replay) +- [ ] Add Uptime Kuma monitor for the tailer service +- [ ] Roll out to production via Ansible +- [ ] Decommission v1 auditd dispatcher ## Open Questions -- Which Google Chat space should alerts go to? -- Do we want any metadata in the alert beyond username + source IP + -timestamp (e.g., geolocation lookup on source IP)? -- Do we want a separate "critical" alert path for root logins? +- Host systemd service or containerized? Leaning host service — touches +host-level journal, simpler bind-mount story. Breaks Compose aesthetic but +fits the job. +- Where does cursor state live? (Path, ownership, permissions — probably +`/var/lib/ssh-login-alerter/cursor`.) +- Which Google Chat space — `webadmin` or a dedicated security space? +- Metadata beyond username + source IP + timestamp (e.g., geolocation on +source IP)? +- Separate "critical" alert path for root logins? +- How does Uptime Kuma know the tailer is alive? Heartbeat ping from the +tailer itself, or process check? ## Priority -Higher priority than env file hardening — it's a force multiplier. Even if -the env hardening isn't done yet, knowing about an unauthorized login -immediately limits blast radius. +Still higher priority than env file hardening. v1's silent drops mean we +currently have *no* working alert on the `pbsdeploy` account — the +force-multiplier argument is sharper now than it was at project inception. +...sent from Jenny & Travis ...sent from Jenny & Travis \ No newline at end of file