pbs-projects/Tech/Projects/ssh-login-alerting.md

project	type	status	path	tags	created	updated
ssh-login-alerting	project-plan	active	Tech/Projects	pbs security ssh journald python n8n monitoring	2026-04-19	2026-04-21

SSH Login Alerting

Goal

Get notified in Google Chat the moment anyone SSHes into a PBS server. Early warning system for compromised SSH keys or unauthorized access — especially important now that pbsdeploy exists as a passphrase-less key-based account.

Architecture

sshd → journald → Python sdjournal tailer → POST to n8n webhook → n8n formats + sends to Google Chat

Design Decisions

• journald over auditd — v1 attempted auditd → shell dispatcher → regex → webhook and was abandoned after silent event drops and two hours of failed debugging. Root cause: auditd emits multiple records per logical event with format variations that break line-by-line regex parsing. Journald is one-line-per-event with structured metadata (_SYSTEMD_UNIT=ssh.service), which eliminates the multi-record coalescing problem entirely.
• Python sdjournal tailer, not pam_exec — pam_exec would give fully structured data with zero regex via PAM env vars (PAM_USER, PAM_RHOST, etc.), but touches the SSH auth path. Explicit tradeoff: accept one remaining regex pattern in exchange for zero risk to login infrastructure. If the tailer crashes, SSH keeps working.
• Cursor persistence solves the silent-drop problem — systemd.journal.Reader.seek_cursor() with cursor saved to disk after each successful webhook POST gives exactly-once delivery across restarts, crashes, and n8n outages. This was the actual failure mode of v1.
• Regex is contained and testable — one pattern against sshd's MESSAGE field (stable across OpenSSH versions for ~10 years), in one Python function, with unit tests against fixture strings. When it breaks, fix is one line in one place.
• n8n handles notification routing — unchanged from v1. Server fires events; n8n formats and delivers.
• Alert on every login for v1 — unchanged. Tune later if noise becomes a problem.
• Scope: all accounts, all PBS servers — unchanged. Staging and production, every user.

Why This Matters

• pbsdeploy SSH key has no passphrase. If it leaks, the attacker's first move is SSHing in.
• Linode account is already behind 2FA, so LISH console access isn't the main threat vector.
• SSH is the primary attack surface — we need eyes on it.

Next Steps

• Decide deployment model: host systemd service vs containerized (leaning host systemd — simpler, more reliable for journal access)
• Write Python tailer using systemd.journal.Reader with add_match(_SYSTEMD_UNIT="ssh.service"), cursor persistence, and HTTP POST to n8n
• Unit test the MESSAGE regex against fixture strings (accepted publickey, accepted password, failed attempts, invalid user)
• Deploy to staging, test cursor behavior across restarts
• Build n8n workflow: webhook trigger → format message → send to Google Chat
• End-to-end test on staging (SSH in, kill tailer, SSH in again, restart tailer, confirm replay)
• Add Uptime Kuma monitor for the tailer service
• Roll out to production via Ansible
• Decommission v1 auditd dispatcher

Open Questions

• Host systemd service or containerized? Leaning host service — touches host-level journal, simpler bind-mount story. Breaks Compose aesthetic but fits the job.
• Where does cursor state live? (Path, ownership, permissions — probably /var/lib/ssh-login-alerter/cursor.)
• Which Google Chat space — webadmin or a dedicated security space?
• Metadata beyond username + source IP + timestamp (e.g., geolocation on source IP)?
• Separate "critical" alert path for root logins?
• How does Uptime Kuma know the tailer is alive? Heartbeat ping from the tailer itself, or process check?

Priority

Still higher priority than env file hardening. v1's silent drops mean we currently have no working alert on the pbsdeploy account — the force-multiplier argument is sharper now than it was at project inception.

...sent from Jenny & Travis

...sent from Jenny & Travis