Env File Hardening

Goal

Remove the easy path to PBS secrets. Right now .env files in /opt/docker are readable by anyone with filesystem access — including pbsdeploy. If the deploy key is compromised, an attacker can cat .env and grab every API key and password instantly.

Goal: pbsdeploy can deploy containers without ever being able to read the secrets they use.

Threat Model

Primary risk: pbsdeploy SSH key leaks → attacker SSHes in → reads .env files → owns every service.
Mitigation goal: eliminate the trivial read path. Raise the bar so an attacker would need to escalate to root (no sudo on pbsdeploy), pivot accounts, or break out of a container.
Accepted residual risk: Determined attackers with deep access can still cause damage. This is about removing low-hanging fruit, not achieving zero risk.

Two-Phase Approach

Phase 1: Filesystem Permissions (80/20 win)

Lock .env files down to root:root 0600. The Docker daemon runs as root, so docker compose still reads them fine when processing compose files. pbsdeploy runs docker compose up, which talks to the daemon via socket — pbsdeploy itself never reads .env directly.

Net effect: the trivial cat .env path is gone. No sidecar needed, no app code changes, no runtime complexity.

Phase 2: Encryption at Rest on the Server (defense in depth, deferred)

Currently secrets are encrypted in the wp-i repo via ansible-vault, but land as plaintext on the server filesystem. Phase 2 keeps them encrypted on disk and decrypts only at container start (via sidecar or similar).

Protects against: backup leaks, Linode snapshot exfil, accidental exposure, offline decrypt of historical filesystem copies.

Not urgent because: secrets are already encrypted in the repo, Linode account is behind 2FA, no compliance driver. This is "nice to have when refactoring," per earlier conversation on the topic.

Design Decisions

Phase 1 is just filesystem perms — no sidecar, no Docker secrets, no app changes. Docker daemon reads .env as root; pbsdeploy doesn't need to.
Phase 2 uses sidecar pattern when we get there — tiny init container decrypts at runtime, injects env vars into main container, zero app code changes. Compose secrets ruled out because it requires refactoring every app to read from /run/secrets/.
External secret manager (Vault, etc.) ruled out — overkill for self-hosted prod scale. Revisit only if moving to Kubernetes.
Ansible stays the deploy tool throughout — writes .env files as root with tight perms. pbsdeploy has no sudoers entry, so it can't read them back.
Staging first for both phases.

How the Permissions Work (Phase 1)

Ansible SSHes in as pbsdeploy → uses become: yes to elevate for file writes → files created as root:root 0600.
pbsdeploy can't cat them (no read perms). Can't sudo cat them either (no sudoers entry).
Docker daemon runs as root → reads .env when processing compose files → injects vars into containers.
Main container sees env vars, doesn't know or care about file perms.

Next Steps (Phase 1)

Audit current state — which services have .env files in /opt/docker, current ownership and perms
Update Ansible role(s) to write .env as root:root 0600 (not deployer group)
Pick a non-critical pilot service on staging
Deploy change to pilot, verify container still works, verify pbsdeploy can't read the file
Roll out to all staging services
Verify nothing breaks over a few deploys
Roll out to production
Remove deployer group read access to .env files (group can still exist for other things if needed)

Next Steps (Phase 2 — later)

Decide on encryption approach: custom sidecar vs. SOPS + age vs. something else
Design sidecar container (Dockerfile, entrypoint, decryption strategy)
Decide where the decryption key lives on the server and how it's protected
Pilot on one staging service
Roll out across stack

Open Questions

Phase 1

Does anything in /opt/docker currently rely on deployer group read access to .env that we'd break by tightening perms?
Are all services using env_file: in compose, or do some hardcode secrets in the compose file itself (which would need separate handling)?

Phase 2 (deferred)

Sidecar implementation: custom shell-based init container vs. off-the-shelf tool like SOPS?
Where does the decryption key live on the server? (If it's in another root-owned file, we're just moving the problem — though one layer deeper is still an improvement.)

SSH Login Alerting — higher priority sibling project. Early warning system for the threat this project mitigates.
PBSII CI/CD — same pbsdeploy / deployer setup on staging; phase 1 changes apply there too.
Previous conversation on secrets managers — concluded that for single-server self-hosted prod, encryption-at-rest on the server is "nice to have," not urgent. Phase 2 is the realization of that future work.

...sent from Jenny & Travis

5.3 KiB Raw Blame History