110 lines
4.1 KiB
Markdown
110 lines
4.1 KiB
Markdown
---
|
|
project: ssh-login-alerting
|
|
type: project-plan
|
|
status: active
|
|
path: Tech/Projects
|
|
tags:
|
|
- pbs
|
|
- security
|
|
- ssh
|
|
- journald
|
|
- python
|
|
- n8n
|
|
- monitoring
|
|
created: 2026-04-19
|
|
updated: 2026-04-21
|
|
---
|
|
# SSH Login Alerting
|
|
|
|
## Goal
|
|
|
|
Get notified in Google Chat the moment anyone SSHes into a PBS server.
|
|
Early warning system for compromised SSH keys or unauthorized access —
|
|
especially important now that `pbsdeploy` exists as a passphrase-less
|
|
key-based account.
|
|
|
|
## Architecture
|
|
|
|
|
|
sshd → journald → Python sdjournal tailer → POST to n8n webhook →
|
|
n8n formats + sends to Google Chat
|
|
|
|
|
|
## Design Decisions
|
|
|
|
- **journald over auditd** — v1 attempted `auditd → shell dispatcher →
|
|
regex → webhook` and was abandoned after silent event drops and two hours
|
|
of failed debugging. Root cause: auditd emits multiple records per logical
|
|
event with format variations that break line-by-line regex parsing.
|
|
Journald is one-line-per-event with structured metadata
|
|
(`_SYSTEMD_UNIT=ssh.service`), which eliminates the multi-record coalescing
|
|
problem entirely.
|
|
- **Python sdjournal tailer, not pam_exec** — pam_exec would give fully
|
|
structured data with zero regex via PAM env vars (`PAM_USER`, `PAM_RHOST`,
|
|
etc.), but touches the SSH auth path. Explicit tradeoff: accept one
|
|
remaining regex pattern in exchange for zero risk to login infrastructure.
|
|
If the tailer crashes, SSH keeps working.
|
|
- **Cursor persistence solves the silent-drop problem** —
|
|
`systemd.journal.Reader.seek_cursor()` with cursor saved to disk after each
|
|
successful webhook POST gives exactly-once delivery across restarts,
|
|
crashes, and n8n outages. This was the actual failure mode of v1.
|
|
- **Regex is contained and testable** — one pattern against sshd's
|
|
`MESSAGE` field (stable across OpenSSH versions for ~10 years), in one
|
|
Python function, with unit tests against fixture strings. When it breaks,
|
|
fix is one line in one place.
|
|
- **n8n handles notification routing** — unchanged from v1. Server fires
|
|
events; n8n formats and delivers.
|
|
- **Alert on every login for v1** — unchanged. Tune later if noise becomes
|
|
a problem.
|
|
- **Scope: all accounts, all PBS servers** — unchanged. Staging and
|
|
production, every user.
|
|
|
|
## Why This Matters
|
|
|
|
- `pbsdeploy` SSH key has no passphrase. If it leaks, the attacker's first
|
|
move is SSHing in.
|
|
- Linode account is already behind 2FA, so LISH console access isn't the
|
|
main threat vector.
|
|
- SSH is the primary attack surface — we need eyes on it.
|
|
|
|
## Next Steps
|
|
|
|
- [ ] Decide deployment model: host systemd service vs containerized
|
|
(leaning host systemd — simpler, more reliable for journal access)
|
|
- [ ] Write Python tailer using `systemd.journal.Reader` with
|
|
`add_match(_SYSTEMD_UNIT="ssh.service")`, cursor persistence, and HTTP POST
|
|
to n8n
|
|
- [ ] Unit test the MESSAGE regex against fixture strings (accepted
|
|
publickey, accepted password, failed attempts, invalid user)
|
|
- [ ] Deploy to staging, test cursor behavior across restarts
|
|
- [ ] Build n8n workflow: webhook trigger → format message → send to Google
|
|
Chat
|
|
- [ ] End-to-end test on staging (SSH in, kill tailer, SSH in again,
|
|
restart tailer, confirm replay)
|
|
- [ ] Add Uptime Kuma monitor for the tailer service
|
|
- [ ] Roll out to production via Ansible
|
|
- [ ] Decommission v1 auditd dispatcher
|
|
|
|
## Open Questions
|
|
|
|
- Host systemd service or containerized? Leaning host service — touches
|
|
host-level journal, simpler bind-mount story. Breaks Compose aesthetic but
|
|
fits the job.
|
|
- Where does cursor state live? (Path, ownership, permissions — probably
|
|
`/var/lib/ssh-login-alerter/cursor`.)
|
|
- Which Google Chat space — `webadmin` or a dedicated security space?
|
|
- Metadata beyond username + source IP + timestamp (e.g., geolocation on
|
|
source IP)?
|
|
- Separate "critical" alert path for root logins?
|
|
- How does Uptime Kuma know the tailer is alive? Heartbeat ping from the
|
|
tailer itself, or process check?
|
|
|
|
## Priority
|
|
|
|
Still higher priority than env file hardening. v1's silent drops mean we
|
|
currently have *no* working alert on the `pbsdeploy` account — the
|
|
force-multiplier argument is sharper now than it was at project inception.
|
|
|
|
...sent from Jenny & Travis
|
|
|
|
...sent from Jenny & Travis |