Update env-file-hardening.md via n8n

Tech/Projects/env-file-hardening.md

@@ -13,7 +13,7 @@ created: 2026-04-19
 updated: 2026-04-19
 ---
 
-# Env File Hardening (Sidecar Pattern)
+# Env File Hardening
 
 ## Goal
 
@@ -36,76 +36,108 @@ accounts, or break out of a container.
 still cause damage. This is about removing low-hanging fruit, not achieving
 zero risk.
 
-## Architecture
+## Two-Phase Approach
 
-**Sidecar injection pattern:**
+### Phase 1: Filesystem Permissions (80/20 win)
 
-- `.env` files stay owned by `root:root` with `600` perms.
-- Ansible writes them via `become: yes` (sudo) — `pbsdeploy` never touches
-the files directly.
-- A tiny init container / sidecar runs as root, reads the locked `.env`,
-and injects secrets into the main container as environment variables.
-- Main containers read env vars exactly as they do today — **no application
-code changes.**
+Lock `.env` files down to `root:root 0600`. The Docker daemon runs as root,
+so `docker compose` still reads them fine when processing compose files.
+`pbsdeploy` runs `docker compose up`, which talks to the daemon via socket
+— `pbsdeploy` itself never reads `.env` directly.
+
+**Net effect:** the trivial `cat .env` path is gone. No sidecar needed, no
+app code changes, no runtime complexity.
+
+### Phase 2: Encryption at Rest on the Server (defense in depth, deferred)
+
+Currently secrets are encrypted in the `wp-i` repo via ansible-vault, but
+land as plaintext on the server filesystem. Phase 2 keeps them encrypted on
+disk and decrypts only at container start (via sidecar or similar).
+
+**Protects against:** backup leaks, Linode snapshot exfil, accidental
+exposure, offline decrypt of historical filesystem copies.
+
+**Not urgent because:** secrets are already encrypted in the repo, Linode
+account is behind 2FA, no compliance driver. This is "nice to have when
+refactoring," per earlier conversation on the topic.
 
 ## Design Decisions
 
-- **Sidecar over Docker Compose secrets** — Compose secrets would require
-refactoring every app (WordPress, n8n, pbs-hub, PBSII layers) to read from
-`/run/secrets/` files instead of env vars. Sidecar gets the security win
-with zero app-level changes.
-- **Sidecar over external secret manager (Vault, etc.)** — overkill for
-homelab/small prod scale. Revisit if we ever move to Kubernetes.
-- **Ansible stays the deploy tool** — still writes `.env` files, just as
-root with tight perms. `pbsdeploy` has no sudo entry, so it can't read them
-back.
-- **Scope: all services on production PBS** — WordPress, n8n, pbs-hub,
-phpMyAdmin, everything in `/opt/docker`. No point hardening one service if
-others still leak.
-- **Staging first** — test on one service on staging, then roll out across
-all services on staging, then move to production.
+- **Phase 1 is just filesystem perms** — no sidecar, no Docker secrets, no
+app changes. Docker daemon reads `.env` as root; `pbsdeploy` doesn't need
+to.
+- **Phase 2 uses sidecar pattern when we get there** — tiny init container
+decrypts at runtime, injects env vars into main container, zero app code
+changes. Compose secrets ruled out because it requires refactoring every
+app to read from `/run/secrets/`.
+- **External secret manager (Vault, etc.) ruled out** — overkill for
+self-hosted prod scale. Revisit only if moving to Kubernetes.
+- **Ansible stays the deploy tool** throughout — writes `.env` files as
+root with tight perms. `pbsdeploy` has no sudoers entry, so it can't read
+them back.
+- **Staging first** for both phases.
 
-## How the Permissions Work
+## How the Permissions Work (Phase 1)
 
 - Ansible SSHes in as `pbsdeploy` → uses `become: yes` to elevate for file
-writes → files created as `root:root 600`.
+writes → files created as `root:root 0600`.
 - `pbsdeploy` can't `cat` them (no read perms). Can't `sudo cat` them
 either (no sudoers entry).
-- Docker daemon runs as root → sidecar container runs as root → can read
-the files and inject into main container.
-- Main container sees env vars, doesn't know or care where they came from.
+- Docker daemon runs as root → reads `.env` when processing compose files →
+injects vars into containers.
+- Main container sees env vars, doesn't know or care about file perms.
 
-## Next Steps
+## Next Steps (Phase 1)
 
-- [ ] Pick a non-critical service on staging to pilot the sidecar pattern
-- [ ] Design the sidecar container (shell script? tiny Go/Python binary?
-off-the-shelf image?)
-- [ ] Update compose file for pilot service to include sidecar
-- [ ] Update Ansible role to write `.env` as `root:root 600` (not
+- [ ] Audit current state — which services have `.env` files in
+`/opt/docker`, current ownership and perms
+- [ ] Update Ansible role(s) to write `.env` as `root:root 0600` (not
 `deployer` group)
-- [ ] Test on staging — verify containers still work, verify `pbsdeploy`
-can't read files
+- [ ] Pick a non-critical pilot service on staging
+- [ ] Deploy change to pilot, verify container still works, verify
+`pbsdeploy` can't read the file
 - [ ] Roll out to all staging services
-- [ ] Verify nothing broke in staging over a few deploys
+- [ ] Verify nothing breaks over a few deploys
 - [ ] Roll out to production
-- [ ] Remove `deployer` group's access to `.env` files (or the group
-entirely, depending on what it ends up being used for)
+- [ ] Remove `deployer` group read access to `.env` files (group can still
+exist for other things if needed)
+
+## Next Steps (Phase 2 — later)
+
+- [ ] Decide on encryption approach: custom sidecar vs. SOPS + age vs.
+something else
+- [ ] Design sidecar container (Dockerfile, entrypoint, decryption strategy)
+- [ ] Decide where the decryption key lives on the server and how it's
+protected
+- [ ] Pilot on one staging service
+- [ ] Roll out across stack
 
 ## Open Questions
 
-- What's the right sidecar implementation? Custom shell-based init
-container vs. existing tool?
-- Which pilot service to start with?
-- Does anything else in `/opt/docker` currently rely on the `deployer`
-group having read access that we'd break by tightening perms?
+### Phase 1
+
+- Does anything in `/opt/docker` currently rely on `deployer` group read
+access to `.env` that we'd break by tightening perms?
+- Are all services using `env_file:` in compose, or do some hardcode
+secrets in the compose file itself (which would need separate handling)?
+
+### Phase 2 (deferred)
+
+- Sidecar implementation: custom shell-based init container vs.
+off-the-shelf tool like SOPS?
+- Where does the decryption key live on the server? (If it's in another
+root-owned file, we're just moving the problem — though one layer deeper is
+still an improvement.)
 
 ## Depends On / Related
 
-- **SSH Login Alerting** — higher priority sibling project. Even before env
-hardening is done, SSH alerts give us early warning if `pbsdeploy` is
-compromised.
-- **PBSII CI/CD** — same `pbsdeploy` / `deployer` setup on staging;
-whatever we do here applies there too.
+- **SSH Login Alerting** — higher priority sibling project. Early warning
+system for the threat this project mitigates.
+- **PBSII CI/CD** — same `pbsdeploy` / `deployer` setup on staging; phase 1
+changes apply there too.
+- **Previous conversation on secrets managers** — concluded that for
+single-server self-hosted prod, encryption-at-rest on the server is "nice
+to have," not urgent. Phase 2 is the realization of that future work.
 
 
 ...sent from Jenny & Travis