---
created: 2026-04-19
path: Sources/Homelab
project: ssh-login-alerting
status: active
tags:
- pbs
- security
- ssh
- journald
- python
- n8n
- monitoring
type: project-plan
updated: 2026-04-21
---

# SSH Login Alerting

## Goal

Get notified in Google Chat the moment anyone SSHes into a PBS server.
Early warning system for compromised SSH keys or unauthorized access —
especially important now that `pbsdeploy` exists as a passphrase-less
key-based account.

## Architecture


sshd → journald → Python sdjournal tailer → POST to n8n webhook →
n8n formats + sends to Google Chat


## Design Decisions

- **journald over auditd** — v1 attempted `auditd → shell dispatcher →
regex → webhook` and was abandoned after silent event drops and two hours
of failed debugging. Root cause: auditd emits multiple records per logical
event with format variations that break line-by-line regex parsing.
Journald is one-line-per-event with structured metadata
(`_SYSTEMD_UNIT=ssh.service`), which eliminates the multi-record coalescing
problem entirely.
- **Python sdjournal tailer, not pam_exec** — pam_exec would give fully
structured data with zero regex via PAM env vars (`PAM_USER`, `PAM_RHOST`,
etc.), but touches the SSH auth path. Explicit tradeoff: accept one
remaining regex pattern in exchange for zero risk to login infrastructure.
If the tailer crashes, SSH keeps working.
- **Cursor persistence solves the silent-drop problem** —
`systemd.journal.Reader.seek_cursor()` with cursor saved to disk after each
successful webhook POST gives exactly-once delivery across restarts,
crashes, and n8n outages. This was the actual failure mode of v1.
- **Regex is contained and testable** — one pattern against sshd's
`MESSAGE` field (stable across OpenSSH versions for ~10 years), in one
Python function, with unit tests against fixture strings. When it breaks,
fix is one line in one place.
- **n8n handles notification routing** — unchanged from v1. Server fires
events; n8n formats and delivers.
- **Alert on every login for v1** — unchanged. Tune later if noise becomes
a problem.
- **Scope: all accounts, all PBS servers** — unchanged. Staging and
production, every user.

## Why This Matters

- `pbsdeploy` SSH key has no passphrase. If it leaks, the attacker's first
move is SSHing in.
- Linode account is already behind 2FA, so LISH console access isn't the
main threat vector.
- SSH is the primary attack surface — we need eyes on it.

## Next Steps

- [ ] Decide deployment model: host systemd service vs containerized
(leaning host systemd — simpler, more reliable for journal access)
- [ ] Write Python tailer using `systemd.journal.Reader` with
`add_match(_SYSTEMD_UNIT="ssh.service")`, cursor persistence, and HTTP POST
to n8n
- [ ] Unit test the MESSAGE regex against fixture strings (accepted
publickey, accepted password, failed attempts, invalid user)
- [ ] Deploy to staging, test cursor behavior across restarts
- [ ] Build n8n workflow: webhook trigger → format message → send to Google
Chat
- [ ] End-to-end test on staging (SSH in, kill tailer, SSH in again,
restart tailer, confirm replay)
- [ ] Add Uptime Kuma monitor for the tailer service
- [ ] Roll out to production via Ansible
- [ ] Decommission v1 auditd dispatcher

## Open Questions

- Host systemd service or containerized? Leaning host service — touches
host-level journal, simpler bind-mount story. Breaks Compose aesthetic but
fits the job.
- Where does cursor state live? (Path, ownership, permissions — probably
`/var/lib/ssh-login-alerter/cursor`.)
- Which Google Chat space — `webadmin` or a dedicated security space?
- Metadata beyond username + source IP + timestamp (e.g., geolocation on
source IP)?
- Separate "critical" alert path for root logins?
- How does Uptime Kuma know the tailer is alive? Heartbeat ping from the
tailer itself, or process check?

## Priority

Still higher priority than env file hardening. v1's silent drops mean we
currently have *no* working alert on the `pbsdeploy` account — the
force-multiplier argument is sharper now than it was at project inception.

...sent from Jenny & Travis

...sent from Jenny & Travis