petal-power/wiki-vault
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: petal-power:52e3e76250cef99e26c0f1ee68370394cfd62b28
Branches Tags
petal-power:main
..
compare: petal-power:9bff04eebf2b9c5e3afb90e7538ab59c72cc7d03
Branches Tags
petal-power:main

No commits in common. "52e3e76250cef99e26c0f1ee68370394cfd62b28" and "9bff04eebf2b9c5e3afb90e7538ab59c72cc7d03" have entirely different histories.
52e3e76250 ... 9bff04eebf

5 changed files with 0 additions and 1133 deletions
Show Stats Expand all files Collapse all files

80
Sources/Dev/2026-05-15-agentic-validation-article-review.md
View File

@ -1,80 +0,0 @@
---
created: '2026-05-15'
path: Sources/Dev
project: zero-check-refactor
status: active
tags:
- claude-code
- automation
- python
- go
type: session-notes
updated: '2026-05-15'
---
## Outcome
Reviewed the GitHub Engineering blog post on validating agentic behavior in non-deterministic environments. Concluded the specific technique (dominator analysis on UI execution graphs) is not directly applicable to the zero-check gauntlet today, but the underlying principle — that agents cannot reliably grade their own homework and need an independent structural validator — is exactly the design premise of zero-check itself. Parking the article's deeper ideas for a future "semantic validation layer" if and when the harness/Lovebug starts delegating substantive code work that needs intent-level checks.
Reference: <https://github.blog/ai-and-ml/generative-ai/validating-agentic-behavior-when-correct-isnt-deterministic/>
## Topics Covered
### What the article proposes
GitHub's Copilot Coding Agent does "computer use" — driving a real VS Code in a containerized environment. Validating this is hard because the agent's path is non-deterministic: loading screens come and go, hotkey vs menu produces the same outcome, timing varies. Traditional record-and-replay or assertion-based tests fail with false negatives.
Their approach, summarized:
1. Record 2–10 successful runs as a Prefix Tree Acceptor (a directed graph of observed states + transitions)
2. Merge them into a unified graph using a three-tier equivalence check (perceptual hash → SSIM → multimodal LLM for semantic equivalence)
3. Apply **dominator analysis** (compiler-theory concept: state A dominates state B if every path from start to B passes through A) to extract the *essential* states — the milestones every successful run must hit
4. Validate new runs by checking they hit the essential states in the right relative order; extra/incidental states are tolerated
Reported result: 100% accuracy on their test suite vs the agent's self-assessment at 82.2%. Recall jumped from 60% to 100%.
### What's genuinely useful
- **Essential vs incidental framing.** Don't validate the exact path; validate whether the agent crossed the checkpoints that matter. Generalizable mental model beyond UI work.
- **"Agents can't grade their own homework."** Their data: agent self-assessment got 0% F1 on distinguishing real bugs from environmental noise. Zero. Independent structural check got 52%. This is the explicit design premise of zero-check — the agent writes the code, an external tool decides whether it's valid.
- **Learning correctness from examples.** Instead of writing assertions, record successful runs and extract the contract automatically. Cheaper in principle, when applicable.
### What's overhyped or doesn't apply
- **The setup is narrow.** Their technique works because UI flows have real graph structure (discrete states, observable transitions). Code generation, refactoring, architectural decisions don't have that structure — the technique doesn't generalize as cleanly as the article implies.
- **"100% accuracy" is single-test-suite marketing.** Controlled experiment on one task type. Not a generalizable claim.
- **The LLM-in-the-loop is soft-pedaled.** They say "no black-box ML judging," but step 2 of the merge calls a multimodal LLM for semantic state equivalence. Scope-limited, but it's there.
- **Bootstrap dependency.** Requires 2–10 *consistently successful* runs to extract dominators. For flaky tasks, getting that bootstrap is itself the hard problem.
### Relevance to zero-check today
Honest read: the article is not actionable for the current gauntlet. Zero-check validates deterministic checks (lint, tests, SAST, secrets, deps) where there's a single right answer per tool. Dominator analysis is overkill for "did ruff pass."
Where the article *would* matter is a hypothetical second layer — validating that the agent's *overall work* met the stated intent, not just that the code passes mechanical checks. Examples:
- "Add JWT auth to the API" — did the agent actually wire JWT into middleware, or just add a TODO comment?
- "Refactor to dependency injection" — did the result actually use DI, or just rename files?
For those questions, lint-pass is necessary but wildly insufficient. That's the territory where structural intent validation gets interesting.
### If/when that second layer becomes real
The article's specific technique (dominators on execution graphs) is the wrong starting point for code-level intent validation — it's the right tool for UI flows, not for diffs. Simpler approaches that fit the existing stack better:
- **Spec-first.** Agent writes a one-paragraph "this is what I'm going to do" before the change. A separate validator LLM reads the diff and judges whether it matches the spec.
- **Test-as-oracle.** Agent writes a test capturing the intent *before* writing the code; the gauntlet runs the test. Essentially harness-enforced TDD.
- **Behavioral fixture.** For tasks with clear input/output shape, record what success looks like and check against it.
All three are in the same family as the article — *external structural check on intent* — but cheaper and better matched to code rather than UI.
## Key Learnings
- The article's core insight (agents need independent structural validators because self-assessment is unreliable) is *the same principle* zero-check operates on. The refactor we did this session is the deterministic-layer version of what the GitHub post does at the UI layer.
- Dominator analysis is a real technique with a clean theoretical basis (compiler control-flow theory) — worth remembering even if not applied today.
- The two-layer mental model is useful: deterministic gauntlet (lint/test/SAST) + semantic intent validator. Today only layer 1 is built. Layer 2 is a future concern, not a current gap.
- Press-release numbers ("100% accuracy") in research blog posts should be read as "worked on our test set" not "general capability."
## Follow-ons
- [ ] Revisit this article if and when harness/Lovebug starts taking on substantive code-generation tasks where mechanical checks aren't enough to know if the work was correct
- [ ] If a semantic validation layer is ever built, evaluate spec-first and test-as-oracle approaches before reaching for graph-based techniques
- [ ] No action on zero-check itself from this review — the in-flight refactor (manifest-driven routing, inconclusive guard) is the right shape for the deterministic layer

122
Sources/Dev/zero-check-refactor.md
View File

@ -1,122 +0,0 @@
---
created: '2026-05-15'
path: Sources/Dev
project: zero-check-refactor
status: active
tags:
- python
- go
- automation
- claude-code
type: project-plan
updated: '2026-05-15'
---
## Goal
Refactor the `zero-check` agentic skill to eliminate silent-fail modes and reduce bash complexity. Move source-file discovery from custom bash `find` invocations to the native discovery engines of each underlying tool (pytest, ruff, semgrep). Add an "inconclusive" guard so an all-skipped run no longer reports green. Update the global Python project scaffold to carry the project-side boundary rules that the gauntlet stops enforcing.
## Locked Decisions
- **Stay on bash.** The orchestrator is dumb glue calling other CLIs — exactly the job bash is built for. No rewrite to Python/Go/just/pre-commit at this stage; revisit only if the gauntlet grows into non-deterministic agent-output validation.
- **Manifest-driven routing.** Bash detects ecosystem via `pyproject.toml`, `requirements*.txt`, `uv.lock`, `go.mod`. Bash does not enumerate source files.
- **Native discovery.** ruff, semgrep, pytest, go test all walk their own trees and respect project-level exclude config.
- **Pytest exit code 5 → skipped (exit 2).** "No tests collected" is an honest skip, not a vacuous pass.
- **Inconclusive guard.** A run where every check skipped is `overall: failed` with `inconclusive: true` in `results.json`.
- **Boundary rules live in the project.** Non-standard venv names (`env/`, `app_env/`) and other project-specific exclusions belong in `pyproject.toml [tool.pytest.ini_options] norecursedirs`, not in the gauntlet.
- **Three-state exit code contract preserved.** `0 = passed`, `1 = failed`, `2 = skipped`. Sub-scripts continue to honor this; orchestrator continues to route on it.
## Open Items
- [ ] Decide whether to add a JSON-schema validator for `results.json` (defensive, but probably overkill at current scope)
- [ ] Decide whether to add a `--strict` flag to `run-all.sh` that fails on any skip, not just all-skip (probably not — defeats the point of skipped-as-a-state)
- [ ] Smoke-test the refactor against at least one Python project, one Go project, and one polyglot/mixed project before adopting as default
## Phases
### Phase 1 — Drop in the refactored scripts
Replace the six scripts in `~/zero-check-pipeline/validate/` with the refactored versions drafted in chat. Files:
- `run-all.sh` — adds inconclusive guard, adds `inconclusive` field to `results.json`
- `secrets.sh` — unchanged (gitleaks already does its own tree walk)
- `lint.sh` — manifest-routed, source-file probe removed
- `sast.sh` — manifest-routed, source-file probe removed
- `deps.sh` — minor normalization (find pattern → `-print -quit` for consistency)
- `tests.sh` — manifest-routed, pytest exit 5 mapped to skipped, no source-file probe
Tasks:
- [ ] Back up current scripts: `cp -r ~/zero-check-pipeline/validate ~/zero-check-pipeline/validate.bak`
- [ ] Drop in the six new scripts from the chat artifacts
- [ ] `chmod +x ~/zero-check-pipeline/validate/*.sh`
- [ ] `bash -n` syntax-check each (already verified clean in draft)
### Phase 2 — Smoke tests
Validate the refactor against real projects before declaring it the default:
- [ ] Run against herbylab MCP server (Python + uv): expect lint/sast/deps/tests all pass, secrets clean
- [ ] Run against a Go project (pick one from gitea): expect Go branches exercise
- [ ] Run against an empty scratch dir: expect `inconclusive: true`, overall failed
- [ ] Run against a Python project with no tests yet: expect `tests` → skipped, not green
- [ ] Confirm `results.json` shape is unchanged for downstream consumers (just gains the `inconclusive` field)
### Phase 3 — Global Python scaffold update
Move the project-side boundary rules into the default `pyproject.toml` template so every new Python project carries them.
- [ ] Locate the current Python scaffold (uv init template / cookiecutter / personal snippets — wherever new projects start from)
- [ ] Add this block to the canonical scaffold:
```toml
[tool.pytest.ini_options]
norecursedirs = [
"env",
"app_env",
".venv",
"venv",
"node_modules",
"build",
"dist",
".git",
]
```
- [ ] Document the rationale inline (one-line comment: "Excludes for tools that walk the tree; keep gauntlet agnostic")
- [ ] If multiple scaffolds exist (e.g., one for FastMCP servers, one for plain libraries), update all of them
- [ ] Backfill the snippet into existing active Python projects on the tower as opportunity arises — not a forced migration
### Phase 4 — SKILL.md hygiene
The skill file at `~/.claude/skills/zero-check/SKILL.md` does not need to change — the contract (run-all.sh → results.json → retry loop) is preserved. But it's worth a once-over:
- [ ] Verify the retry-loop guidance still makes sense with the inconclusive state (an inconclusive failure should not trigger a fix loop — there's nothing to fix; it should escalate to the human)
- [ ] Consider adding: "If `inconclusive: true` in results.json, STOP and ask the user — do not retry." Inconclusive means the gauntlet did not validate; retrying won't change that.
## Notes
### What changed and why
The original gauntlet had a class of silent failures rooted in bash trying to mimic the discovery engines of the tools it orchestrates. Two specific failure modes:
1. **SIGPIPE under `pipefail`.** `find ... | head -1` on dense trees could die with exit 141, killing the script before any output. The `-print -quit` workaround was already in place but only on some scripts.
2. **Vacuous passes.** A project with no `.py` files exited 0 from lint/sast/tests, contributing to a green overall result even though nothing was checked.
The refactor addresses both by removing the bash-side file walking entirely (mode 1 cannot occur on a manifest lookup) and by adding the inconclusive guard (mode 2 is now flagged).
### Why not rewrite in Python
Considered and rejected for now. The gauntlet's current job is shelling out to other CLI tools and aggregating exit codes — bash is genuinely correct for this. A Python rewrite would buy real value only if the gauntlet evolves into something more sophisticated (e.g., LLM-driven semantic validation of agent output, per the GitHub article on agentic behavior validation). Revisit then.
### Reference: refactored scripts
The six refactored scripts were drafted in conversation and produced as downloadable artifacts. Drop-in replacements for the files in `~/zero-check-pipeline/validate/`. The diff against the originals is:
- All five sub-scripts: source-file `find` calls replaced with manifest detection
- `tests.sh`: pytest exit code 5 explicitly mapped to skipped state
- `lint.sh`, `tests.sh` (Go branch): `cd $gomod_dir && <tool> ./...` pattern adopted
- `run-all.sh`: inconclusive guard added after the check loop; `inconclusive` field added to `results.json` schema
### Reference: GitHub article (parking lot)
Pending review: <https://github.blog/ai-and-ml/generative-ai/validating-agentic-behavior-when-correct-isnt-deterministic/>. Relevant to the question of whether the gauntlet should grow beyond deterministic checks. Decision deferred to a separate session.

298
Sources/Homelab/2026-05-15-postgres-consolidation-phase0-recon.md
View File

@ -1,298 +0,0 @@
---
created: '2026-05-15'
path: Sources/Homelab
project: postgres-consolidation-reflection-layer
status: active
tags:
- ob1
- mcp
- pgvector
- embeddings
- session-note
type: session-note
updated: '2026-05-15'
worker: 2
---
# Postgres Consolidation — Worker 2, Phase 0 recon
Snapshot of the homelab Postgres instance before any consolidation
work. Recorded from `homelab-postgres` (pgvector/pgvector:pg17),
container in `/opt/projects/docker/postgres/`, port `127.0.0.1:5433`,
volume `ob1_data` (external).
## Postgres instance
- Container: `homelab-postgres` (image `pgvector/pgvector:pg17`)
- Volume: `ob1_data` (external — survives stack down)
- Network: `homelab` (alias `homelab-postgres`, also `postgres`)
- Healthy, up 40 hours.
- POSTGRES_DB default = `openbrain`.
## Databases on instance
| Name | Owner | Size | pgvector | Notes |
|-------------|-----------------|---------|----------|-------|
| `herbylab` | `herbylab_user` | 7958 kB | NO | Holds the herby-dev artifact store |
| `openbrain` | `postgres` | 8838 kB | YES (0.8.2) | OB1 captures live here |
| `trellis` | `trellis_user` | 8262 kB | NO | Trellis MCP threads/events |
| `postgres` | `postgres` | 7478 kB | - | default; not touched |
No other user databases. Plan's "instance hosts only wiki, ob1, trellis"
holds — under the actual names `herbylab` / `openbrain` / `trellis`.
## Existing roles
- `postgres` — Superuser, Create role, Create DB, Replication, Bypass RLS
- `herbylab_user` — owns `herbylab.herbylab` schema
- `trellis_user` — owns `trellis.trellis` schema
No `lovebug` role exists. No `ob1_mcp` / `vault_mcp` / `trellis_mcp`
service users yet — these come in Phase 2.
## Schemas, tables, row counts
### `herbylab` DB
- Schemas: `herbylab` (owner `herbylab_user`), `public`.
- Tables in `herbylab`:
- `projects` — 0 rows (8192 B; one empty heap page)
- `artifacts` — 0 rows (8192 B; one empty heap page)
- `alembic_version` — 1 row (`9c7e3f1a2b4d`)
- Custom enums: `herbylab.project_status`, `herbylab.artifact_type`, `herbylab.project_domain`.
- Triggers / functions: `projects_notify` (NOTIFY on row change), `artifacts_set_updated_at`.
- No vector columns.
**Why empty (verified, not a missed lookup):** the schema was created
by alembic migration `9c7e3f1a2b4d` on the unmerged branch
`claude/competent-knuth-de9483` (commit `90bc6a9 Phase 1: herbylab
Postgres schema infrastructure`). That branch also has commits
`8e408c5 Phase 2: polymorphic artifact tools backed by Postgres +
vault` and `78ec094 Phase 3: Trello projection via n8n` which would
have wired writes into the schema, but **the branch is not merged to
`main`.**
The deployed `herbydev-mcp` runs `main`'s code, which has zero DB
access:
- `pyproject.toml` declares no DB driver (no psycopg / sqlalchemy /
asyncpg).
- `mcp_server/tools/memory.py` and `mcp_server/tools/tasks.py` are
empty stub files marked "Phase 2. Do not implement until workflow
patterns are established."
- `mcp_server/tools/vault.py` writes via `core/git_writer.py` →
filesystem + git commit + push to gitea. No database insertion.
- `HERBYLAB_DATABASE_URL` is plumbed through compose env but never
read by the running code.
So the DB schema exists from an integration-test run of the unmerged
branch's migration; no code path on `main` ever populates it.
Implication for consolidation: the `wiki` schema in `petalbrain` is
genuinely empty content-wise. Worker 4's vault-indexing source is the
filesystem (`/opt/projects/wiki-vault/`), not a `wiki.notes` /
`wiki.artifacts` table. If `competent-knuth-de9483` lands later, its
artifact tooling will need to be re-pointed at `petalbrain.wiki.*`
instead of `herbylab.herbylab.*`, and the Phase 4 reflection design
should account for `wiki.artifacts` becoming a populated source over
time.
### `openbrain` DB
- Schemas: `public` only.
- Tables in `public`:
- `thoughts` — 70 rows (live; daemons writing today)
- pgvector 0.8.2 in `public`.
`public.thoughts` shape (relevant columns):
```
id bigint PK (nextval thoughts_id_seq)
content text NOT NULL
embedding vector(768)
metadata jsonb DEFAULT '{}'
user_id text NOT NULL DEFAULT 'travis'
created_at timestamptz DEFAULT CURRENT_TIMESTAMP
updated_at timestamptz DEFAULT CURRENT_TIMESTAMP
```
Existing indexes on `public.thoughts`:
- `thoughts_pkey` (id)
- `idx_thoughts_created_at` (created_at DESC)
- `idx_thoughts_embedding_hnsw` — **hnsw (embedding vector_cosine_ops) WITH (m=16, ef_construction=64)** — already matches the locked Phase 4 spec for `public.embeddings`.
- `idx_thoughts_fingerprint` — UNIQUE (md5(content), user_id)
- `idx_thoughts_metadata` — gin (metadata)
- `idx_thoughts_user_created` (user_id, created_at DESC)
- `idx_thoughts_user_id` (user_id)
Additional objects:
- Function `match_thoughts(query_embedding vector, match_threshold float, match_count int, filter jsonb, p_user_id text)` — bare-`vector` param (no dim), so dimension changes flow through unchanged.
- Trigger `thoughts_updated_at` -> `update_updated_at()`.
Live writer state: 70 rows total, 0 with NULL embedding, 2 still flagged `metadata->>'pending' = 'true'` (enricher backlog). Most recent row id 255 at `2026-05-16 02:07:23 UTC`. Watcher and enricher daemons confirmed live.
### `trellis` DB
- Schemas: `trellis` (owner `trellis_user`), `public`.
- Tables in `trellis`:
- `threads` — 13 rows
- `events` — 50 rows
- `tags` — 42 rows
- `thread_tags` — 55 rows
- `alembic_version` — 1 row
- No vector columns.
## OB1 embedding model
From `ob1-mcp` container env (and `/opt/projects/homelab/ob1-deploy/compose.yml`):
- `EMBEDDING_PROVIDER=ollama`
- `OLLAMA_URL=http://ollama:11434`
- `EMBEDDING_MODEL=nomic-embed-text` (768-d output)
- `CHAT_MODEL=openai/gpt-4o-mini` (mcp side)
- `CHAT_MODEL=llama3.2:3b` (enricher side)
Migration trail:
- `init.sql` originally declared `vector(1536)` for `text-embedding-3-small`.
- `migrations/001_embedding_dim_768.sql` (applied 2026-05-14) dropped the
HNSW index, altered the column to `vector(768)`, recreated the index.
Required `thoughts` to be empty pre-migration.
**Conclusion:** the shared `public.embeddings` table goes in as
`vector(768)` to match `nomic-embed-text`, no model switch needed. HNSW
index spec from Travis's overrides (`vector_cosine_ops, m=16,
ef_construction=64`) matches the existing OB1 convention exactly.
## MCP server connection configurations
| Container | Image | DB host / DB / user | Live? |
|-----------------|--------------------|---------------------|-------|
| `herbydev-mcp` | `herbydev-mcp` | `homelab-postgres:5432/herbylab` as `herbylab_user` | Up 40h |
| `vault-mcp` | `vault-mcp` | **`vault-postgres:5432/herbylab` as `herbylab_user`** | Up 59m |
| `ob1-mcp` | `ob1-mcp` | `homelab-postgres:5432/openbrain` as `postgres` | Up 14h |
| `ob1-watcher` | `ob1-watcher` | (talks to ob1-mcp HTTP, not DB directly) | Up 26h |
| `ob1-enricher` | `ob1-enricher` | (uses `OLLAMA_URL`; DB password set but no direct connect string visible) | Up 14h |
| `trellis-mcp` | `trellis-trellis-mcp` | `homelab-postgres:5432/trellis` as `trellis_user` | Up 40h healthy |
Connection strings live in:
- `vault-mcp` / `herbydev-mcp` — `/opt/projects/homelab/herby-dev/deploy/.env` (`HERBYLAB_DATABASE_URL`)
- `ob1-mcp` / `ob1-enricher` — `/opt/projects/homelab/ob1-deploy/.env` (`POSTGRES_PASSWORD` + per-service `DB_*` env in `compose.yml`)
- `trellis-mcp` — `/opt/projects/homelab/trellis-mcp/` (compose env, `DATABASE_URL`)
**Anomaly:** `vault-mcp` references hostname `vault-postgres` which
does not resolve on the `homelab` network (alias is `homelab-postgres`
+ `postgres`). This is consistent with Travis's mid-flight rename
rollout — the container is currently running OLD code per the
priming, and the connection-string update is part of the unfinished
rollout. Not blocking Phase 1, but Phase 3 must coordinate with the
rename completion.
## Backup destination
**Local-only on herbydev** (Travis decision, 2026-05-15). The NAS
reference in the plan was unintended — disregard it.
- Destination directory: `/opt/backups/postgres-consolidation/`
(does not exist yet; Phase 1 step zero is `mkdir -p` it).
- Local disk on `/`: 14 G free of 97 G (86% used).
- Combined DB size is ~25 MB, so headroom is ample for both Phase 1
bulk dumps and Phase 2-start fresh dumps.
- Format: `pg_dumpall` (full instance) + per-database `pg_dump` in
custom format (`-Fc`), gzipped. Filenames include UTC timestamp.
- No NAS / Tailscale / off-host destination involved.
## Coordination state — running daemons
`ob1-watcher` and `ob1-enricher` are both live and writing to
`openbrain.public.thoughts` (latest write ~5 minutes before this
note). Per priming, the Phase 2 `ALTER TABLE public.thoughts SET
SCHEMA ob1` step is the cutover gate: daemons must be paused, table
moved + ob1.thoughts shape rewrite by Worker 3, then daemons resumed.
Worker 2 stops at the announcement of readiness for that move and
waits.
## Naming reconciliation (plan vs. reality)
Plan vocabulary `wiki / ob1 / trellis` will land as the **schema**
names in the consolidated database, per Travis's locked decisions.
Source DB → target schema mapping:
- `herbylab` DB content → `wiki` schema (owned by `vault_mcp` per
override; was `herby_mcp` in the plan)
- `openbrain` DB content → `ob1` schema; `public.thoughts` becomes
`ob1.thoughts` (preserving `thoughts` vocabulary per override; plan
said `ob1.captures`)
- `trellis` DB content → `trellis` schema (owned by `trellis_mcp`)
Plus shared `public.embeddings` per Phase 4.
**Open: target database name itself.** OB1's database is currently
`openbrain`. The plan says "if OB1's database is the target and is
named appropriately, use in place. Otherwise, rename or create new."
`openbrain` becomes a misleading name once it also holds wiki +
trellis schemas. Worth a deliberate rename (e.g., `homelab`,
`petalbrain`, or similar) before Phase 2 work begins.
## Phase 1 readiness assessment
Ready:
- Source DB sizes are trivial; dumps will be small and fast.
- pgvector already present in target DB (`openbrain`); no extension
install needed.
- Embedding model + dim are confirmed and stable (`nomic-embed-text` /
768) — no re-embed forced.
- Existing HNSW spec on `public.thoughts` matches the locked
`public.embeddings` HNSW spec exactly.
Not ready / needs decision:
- ~~NAS backup destination~~ **Resolved 2026-05-15**: local-only at
`/opt/backups/postgres-consolidation/`. No NAS involved.
- ~~Target database name~~ **Resolved 2026-05-15**: `petalbrain`.
`openbrain` renamed before Phase 2 destructive ops begin.
- **`wiki` schema content scope** — `herbylab` DB schema came from the
unmerged `claude/competent-knuth-de9483` branch's migration
`9c7e3f1a2b4d`; `main`'s deployed MCP doesn't open a DB connection
at all. Wiki content is filesystem-only under
`/opt/projects/wiki-vault/`. Worker 4's vault-indexing source must
walk the filesystem and write into `public.embeddings` directly.
See the herbylab-DB section above for the full evidence trail.
- **Connection-string anomaly in `vault-mcp`** (`vault-postgres`
hostname). Tied to Travis's rename rollout, blocks Phase 3 cutover
for that MCP only. Phase 1/2 unaffected.
Conservative call: gate before Phase 1 until the NAS destination and
target DB name decisions are made.
## Addendum — Phase 2-start fresh backup (Travis directive, 2026-05-15)
**In addition to Phase 1 bulk backups**, Phase 2 must begin with a
fresh `pg_dump` of each source database immediately before the first
destructive operation (no `CREATE SCHEMA`, no `ALTER TABLE`, no
restore — nothing destructive precedes this step).
Rationale: `ob1-watcher` is writing to `public.thoughts` continuously
(latest write ~5 min before this note). Time elapses between Phase 1
dumps and Phase 2 cutover, especially with gating. Any rows captured
in that window would be lost on rollback if Phase 1 dumps are the
only restore point.
Specifics:
- Run fresh `pg_dump` on `openbrain` first — it's the live writer and
the most critical. Then `herbylab` and `trellis` for symmetry.
- Destination: `/opt/backups/postgres-consolidation/` (same as Phase
1), gzipped custom format.
- Filename suffix: `-phase2-start-<UTC-timestamp>` to distinguish from
Phase 1 backups.
- Verify each fresh dump is readable via test restore to a scratch
database (same pattern as Phase 1).
- Only after the three fresh dumps are verified does Phase 2's first
destructive operation run.
This is not a replacement for Phase 1 dumps — Phase 1 remains the
broad safety net (full instance `pg_dumpall` + per-DB granular
dumps). Phase 2-start dumps are the just-before-disruption snapshot
that captures any in-flight writer activity.
Operational note: this is also the natural point to pause
`ob1-watcher` and `ob1-enricher`. Sequence:
1. Pause both daemons (`docker compose stop ob1-watcher ob1-enricher`).
2. Confirm no in-flight writes (`pg_stat_activity` quiet on `openbrain`).
3. Take the fresh per-DB dumps + verify.
4. Begin Phase 2 destructive work.
5. (Worker 3 resumes daemons after `ob1.thoughts` cutover lands.)

252
Sources/Homelab/2026-05-15-postgres-consolidation-phase2a-result.md
View File

@ -1,252 +0,0 @@
---
created: '2026-05-15'
path: Sources/Homelab
project: postgres-consolidation-reflection-layer
status: active
tags:
- ob1
- mcp
- pgvector
- embeddings
- session-note
type: session-note
updated: '2026-05-15'
worker: 2
phase: 2a
---
# Postgres Consolidation — Worker 2, Phase 2a result
Light Phase 2a complete. Disruption-free: no consumer connections
broken, no live writers paused, no schema rename. `openbrain` is still
the live DB name; `petalbrain` rename is Worker 3's cutover.
## What landed
Inside `openbrain` (the existing DB):
### New roles (no grants on `ob1` yet — that schema doesn't exist)
| Role | Schema-level grants | search_path |
|---------------|----------------------------------------------------------------------------|--------------------------------|
| `vault_mcp` | OWNER of `wiki` | `wiki, public` |
| `trellis_mcp` | OWNER of `trellis` | `trellis, public` |
| `ob1_mcp` | USAGE + SELECT on `wiki` and `trellis` | `ob1, public` (ob1 not yet created) |
| `lovebug` | USAGE + SELECT + INSERT/UPDATE/DELETE on `wiki` and `trellis` | `wiki, ob1, trellis, public` |
All four: LOGIN, NOSUPERUSER, NOCREATEDB, NOCREATEROLE, NOREPLICATION,
NOBYPASSRLS, INHERIT.
Default privileges configured so future tables owned by `vault_mcp`
(in `wiki`) and `trellis_mcp` (in `trellis`) inherit the same grant
model automatically.
Credentials at `/opt/backups/postgres-consolidation/credentials.env`
(mode 0600, herbyadmin-only). 64-char hex passwords. For KeePassXC
transcription.
### Schemas in `openbrain`
- `wiki` — restored from Phase 1 `herbylab-pg_dump`, then
`ALTER SCHEMA herbylab RENAME TO wiki`. Contains:
- tables `alembic_version` (1 row), `artifacts` (0 rows),
`projects` (0 rows)
- sequences `artifacts_id_seq`, `projects_id_seq` (linked to
IDENTITY columns)
- enums `artifact_type`, `project_status`, `project_domain`
- functions `artifacts_set_updated_at`, `notify_project_change`
- triggers `artifacts_updated_at_bump`, `projects_notify`
- FK constraint `projects.artifact_id` → `artifacts.id`
- all OID references updated transparently across the rename
- `trellis` — restored from Phase 1 `trellis-pg_dump` (same name in
source and target, no rename). Contains:
- tables `alembic_version` (1), `events` (50), `tags` (42),
`thread_tags` (55), `threads` (13)
- sequences `events_id_seq`, `tags_id_seq`, `threads_id_seq`
(OWNED BY linked)
- FK chain across `events ⇒ threads`, `thread_tags ⇒ threads`,
`thread_tags ⇒ tags`, `threads.parent_id ⇒ threads`
- `public` — **untouched**. `public.thoughts` still owned by
`postgres`, still receiving writes from `ob1-watcher` (75 rows at
end of Phase 2a, vs 74 at Phase 1 dump start, vs 70 at Phase 0
recon — `ob1-watcher` is healthy). Worker 3 moves this to `ob1`.
- `public.embeddings` — **new, empty**. Shared cross-schema embedding
store per locked spec:
```
public.embeddings (
id BIGSERIAL PRIMARY KEY,
source_schema TEXT NOT NULL,
source_table TEXT NOT NULL,
source_id BIGINT NOT NULL,
embedding VECTOR(768) NOT NULL,
embedding_model TEXT NOT NULL,
embedded_at TIMESTAMPTZ NOT NULL DEFAULT now(),
UNIQUE (source_schema, source_table, source_id, embedding_model)
)
```
Indexes:
- `embeddings_pkey` (id)
- `embeddings_source_uniq` UNIQUE (source_schema, source_table,
source_id, embedding_model)
- `idx_embeddings_hnsw` HNSW (embedding vector_cosine_ops) WITH
(m=16, ef_construction=64) — matches the existing
`idx_thoughts_embedding_hnsw` convention exactly
- `idx_embeddings_source` (source_schema, source_table, source_id)
— for orphan cleanup + source-aware queries
- `idx_embeddings_model` (embedding_model) — for the multi-model
coexistence case the plan calls out
Owned by `postgres` (neutral shared infrastructure). All four
service users have SELECT+INSERT+UPDATE+DELETE on the table and
SELECT+USAGE on the `embeddings_id_seq` sequence. No FK — orphan
cleanup is a separate periodic job (out of scope here per plan).
## What Phase 2a did NOT do
- Did not rename `openbrain` → `petalbrain`. That's a destructive
cutover (live consumers must disconnect for `ALTER DATABASE
RENAME`). Deferred to Worker 3.
- Did not create the `ob1` schema. Worker 3 owns the
`ALTER TABLE public.thoughts SET SCHEMA ob1` step + accompanying
shape rewrite.
- Did not touch `public.thoughts` in any way. Still in `public`,
still writable, still being written to.
- Did not pause `ob1-watcher` or `ob1-enricher`. Both still live.
- Did not modify any MCP container connection strings or env files.
All MCPs still connect to `openbrain` as before.
- Did not touch the source `herbylab` or `trellis` databases. Both
still exist on the instance with their original content; cleanup
is Phase 8 of the plan.
## Verification done
### Row counts in openbrain match dump-source counts exactly
| Schema/Table | Live count | Dump-source count |
|---------------------------|-----------:|------------------:|
| `wiki.projects` | 0 | 0 (herbylab) |
| `wiki.artifacts` | 0 | 0 (herbylab) |
| `wiki.alembic_version` | 1 | 1 (herbylab) |
| `trellis.threads` | 13 | 13 |
| `trellis.events` | 50 | 50 |
| `trellis.tags` | 42 | 42 |
| `trellis.thread_tags` | 55 | 55 |
| `trellis.alembic_version` | 1 | 1 |
| `public.thoughts` | 75 | (live; was 74 at dump time) |
| `public.embeddings` | 0 | (new) |
Source `herbylab` DB and source `trellis` DB confirmed unchanged.
### Permission boundary smoke tests (positive isolation per plan Phase 3)
All ran via direct postgres connection on
`postgresql://<role>:<pw>@localhost:5432/openbrain`:
| Test | Expected | Result |
|--------------------------------------------------|----------|--------|
| `vault_mcp` SELECT `wiki.artifacts` | succeed | OK |
| `vault_mcp` INSERT `trellis.threads` | DENY | `ERROR: permission denied for schema trellis` |
| `trellis_mcp` SELECT `wiki.artifacts` | DENY | `ERROR: permission denied for schema wiki` |
| `ob1_mcp` SELECT `wiki.artifacts` + `trellis.threads` | succeed | OK (counts 0 + 13) |
| `ob1_mcp` INSERT `wiki.artifacts` | DENY | `ERROR: permission denied for table artifacts` |
| `vault_mcp` INSERT `public.embeddings` | succeed | OK (id 1) |
| `ob1_mcp` INSERT `public.embeddings` | succeed | OK (id 2) |
| `trellis_mcp` INSERT `public.embeddings` | succeed | OK (id 3) |
| `lovebug` INSERT `public.embeddings` | succeed | OK (id 4) |
Smoke-test rows (4) deleted after verification;
`public.embeddings` back to 0 rows.
## Surprises / notes
- **Sequence ownership followed table.** First pass on `ALTER OWNER`
failed with `cannot change owner of sequence "artifacts_id_seq" —
Sequence "artifacts_id_seq" is linked to table "artifacts"`. The
herbylab dump uses `GENERATED ALWAYS AS IDENTITY`, whose sequence
is owned by the table and cannot be reassigned independently.
Fixed by excluding `relkind='S'` from the table loop; sequence
ownership now follows the table automatically. Trellis sequences
use SERIAL/OWNED BY which has the same behavior. Caught + handled
in one retry; final state correct.
- **`ob1_mcp.search_path = ob1, public`** is set now even though the
`ob1` schema doesn't exist yet. Once Worker 3 creates the schema
in the same DB (whatever its final name), unqualified table refs
from `ob1-mcp`'s connection will resolve there automatically.
- **No `vault-mcp` env update** was needed for Phase 2a. The current
`HERBYLAB_DATABASE_URL` in `vault-mcp` points at hostname
`vault-postgres` (still anomalous — part of Travis's rename
rollout) and DB `herbylab`. Neither is wired into the consolidated
state yet. Phase 3 of the original plan handles MCP connection
cutover.
## Worker 3 readiness assessment
Worker 3's cutover window is the next destructive step. From the
priming, Worker 3 must:
1. Pause `ob1-watcher` + `ob1-enricher` + `ob1-mcp`.
2. Take Phase 2-start fresh `pg_dump` of `openbrain`, `herbylab`,
`trellis` to `/opt/backups/postgres-consolidation/` with suffix
`-phase2-start-<UTC-timestamp>`. Verify via test restore.
3. `ALTER DATABASE openbrain RENAME TO petalbrain` (now safe — no
live connections).
4. Connect to `petalbrain` and `ALTER TABLE public.thoughts SET
SCHEMA ob1`, plus all dependent objects (indexes are intra-table
so they follow; the `match_thoughts` function references
`thoughts` directly without schema qualification so search_path
must include `ob1` when ob1_mcp calls it — already set; the
`thoughts_updated_at` trigger follows the table; `update_updated_at()`
function lives in `public` and should be moved or be referenced
cross-schema).
5. Create `ob1` schema if `SET SCHEMA` didn't auto-create it
(`SET SCHEMA` does create the schema if it didn't exist — but
actually it requires the schema to pre-exist; Worker 3 should
`CREATE SCHEMA ob1 AUTHORIZATION ob1_mcp` first).
6. ob1-specific grants: `ob1_mcp` owns `ob1.*`; `lovebug` gets full
RW; `vault_mcp` and `trellis_mcp` get nothing.
7. Backfill embeddings: for each of the 75-ish rows in
`ob1.thoughts`, insert into `public.embeddings` with
`source_schema='ob1'`, `source_table='thoughts'`, `source_id=<row
id>`, `embedding=<existing embedding column value>`,
`embedding_model='nomic-embed-text'`.
8. Decide: keep the `ob1.thoughts.embedding` column post-backfill
(redundant with `public.embeddings`) or drop it. Plan says all
embeddings live in `public.embeddings`; suggests dropping. But
that changes the row shape for `ob1-mcp`'s code. Worker 3 call.
9. Rewrite `ob1-mcp`'s `add_thought` and `search_thoughts` for the
new shape (target table `ob1.thoughts`, embedding pushed to
`public.embeddings` separately, search joins).
10. Update connection strings on `ob1-mcp`, `ob1-watcher`,
`ob1-enricher` from DB `openbrain` to DB `petalbrain`. Watcher
talks to ob1-mcp HTTP only, so its env may not need DB host
change — verify.
11. Update `trellis-mcp` and `herbydev-mcp` / `vault-mcp` connection
strings from their current DBs (`trellis`, `herbylab`) to the
consolidated `petalbrain` with their service-user credentials
from `credentials.env`.
12. Resume `ob1-watcher` + `ob1-enricher`.
13. Verify: row counts, semantic search query roundtrip, NOTIFY
trigger on `wiki.projects` still emits.
Phase 2a outputs Worker 3 will need:
- Service user passwords in
`/opt/backups/postgres-consolidation/credentials.env`.
- Phase 1 backup set in `/opt/backups/postgres-consolidation/` plus
the rollback runbook there.
- This session note.
## Holding pattern
`ob1-watcher` and `ob1-enricher` continue writing to
`public.thoughts`. Each minute that passes, the gap between the Phase
1 dump's `thoughts` snapshot (74) and the cutover-time count grows.
Phase 2-start fresh dump shrinks that gap to seconds when Worker 3
runs. Until then: no degradation, no risk; just normal capture.
Worker 2 standing by. No further actions until Worker 3 spawns or
Travis redirects.

381
Sources/Homelab/postgres-consolidation-ob1-reflection-layer.md
View File

@ -1,381 +0,0 @@
---
created: '2026-05-15'
path: Sources/Homelab
project: postgres-consolidation-reflection-layer
status: active
tags:
- ob1
- mcp
- pgvector
- embeddings
- automation
- claude-code
- dispatch
type: project-plan
updated: '2026-05-15'
---
# Postgres Consolidation & OB1 Reflection Layer
## Goal
Two coupled outcomes:
1. **Consolidate** the three project databases (`wiki`, `ob1`, `trellis`) on
the shared Postgres instance into a single database with three schemas,
each owned by a dedicated MCP service user.
2. **Build the OB1 reflection layer**: a two-layer model in `ob1` (captures
+ reflections) with a daily cron job that synthesizes generalizable
lessons from agent session content, plus vault indexing so semantic
search spans vault notes, captures, and reflections uniformly.
These are paired because the reflection loop is fundamentally cross-schema
(citing vault notes from captures from reflections in one query). The
consolidation is the foundation that makes the reflection design clean.
## Locked Decisions
### Database consolidation
- One Postgres database, three schemas: `wiki`, `ob1`, `trellis`.
- Migration direction: **OB1's existing database is the target.** Add
`wiki` and `trellis` schemas to it. Preserves OB1's existing pgvector
setup if installed.
- Target database name, schema-prefix-rewrite details, and final role
names: deferred to execution. Pick deliberate names at that stage.
- Nothing else lives on the Postgres instance — old databases can be
dropped cleanly after verification.
- Downtime during cutover is acceptable. No parallel-run period needed.
### MCP service users (conservative v1)
- `herby_mcp` — owns `wiki.*`. No reads on `ob1` or `trellis`.
- `ob1_mcp` — owns `ob1.*`. **Reads** on `wiki.*` and `trellis.*`
(needed for reflection generation).
- `trellis_mcp` — owns `trellis.*`. No reads on `wiki` or `ob1`.
- `lovebug` — read-write across all three schemas (preserves current
broad-access behavior; tightening deferred).
- Each MCP server connects with its own credentials and its own
`search_path` so unqualified table references resolve to its own schema.
- Permission model revisits after reflection ships and real cross-schema
read patterns are observed.
### Embeddings table
- Shared `public.embeddings` table with pointers back to source rows:
```
public.embeddings (
id bigserial PRIMARY KEY,
source_schema text NOT NULL,
source_table text NOT NULL,
source_id bigint NOT NULL,
embedding vector(<dim>) NOT NULL,
embedding_model text NOT NULL,
embedded_at timestamptz NOT NULL DEFAULT now(),
UNIQUE (source_schema, source_table, source_id, embedding_model)
)
```
- One table for all embeddings across all content types. Cross-source
semantic search becomes one `ORDER BY embedding <-> query_vec` query.
- The unique constraint allows running multiple embedding models in
parallel for experimentation later without conflicting rows.
- Orphan rows possible (no FK because `source_table` varies); cleaned
up by a periodic job (defer; not Phase 1 work).
### Two-layer OB1 schema
- `ob1.captures` — raw layer. Holds Dispatch transcripts, Claude Code
session content, claude.ai chat captures. Written by the capture path;
never rewritten.
- `ob1.reflections` — compiled layer. Holds synthesized lessons generated
by the daily reflection job. Structured object: lesson text, citations
(jsonb array of source pointers), tags, confidence score, generation
metadata (model, date, source window).
- Mirrors the wiki's Sources/Wiki two-layer model. Same uniform pattern
across both systems: raw layer is the substrate, compiled layer is
regenerable.
- Both layers embedded the same way via `public.embeddings`. Retrieval
can scope to one or both via `source_table` filter.
### Reflection job
- **Trigger:** daily cron, 3am local. Off-hours, low contention.
- **Inputs:** all three streams:
- Dispatch session transcripts (Phase 7 of original ob1 plan; pre-compact
hook target — may need to coordinate timing)
- Claude Code session logs from `~/.claude/projects/` (ob1-jsonl-watcher
project is the existing capture mechanism)
- claude.ai chat captures (new path — needs design; see Open Items)
- **Output:** structured reflection records:
```
ob1.reflections (
id bigserial PRIMARY KEY,
lesson text NOT NULL,
citations jsonb NOT NULL, -- [{schema, table, id, relevance}, ...]
tags text[],
confidence float, -- model-reported or heuristic
source_window tstzrange, -- what time window this synthesized
generated_at timestamptz NOT NULL DEFAULT now(),
generated_by text NOT NULL, -- model name + version
job_run_id uuid -- groups reflections from one run
)
```
- **Run-as identity:** `ob1_mcp` user (which has the reads it needs on
`wiki` and `trellis` under v1 permissions).
- **Idle behavior:** if a day has no significant activity, the job writes
no reflections and logs a no-op. No "no significant activity" placeholder
rows.
### Embedding model
- Whatever OB1 currently uses for its own captures. Vault indexing,
capture embedding, and reflection embedding all use the same model
so vectors live in the same space.
- Phase 0 confirms which model OB1 is on.
- Switching the model later is a deliberate full re-embed operation, not
a config flip. The `embedding_model` column on `public.embeddings`
supports incremental migration when that day comes.
### Vault indexing
- In scope for this plan. Without it, reflections can't cite vault notes
semantically.
- Backfill: one-time pass over all existing `wiki.notes` rows, embed
each, insert into `public.embeddings`.
- Incremental: `save_vault_note` in the herby MCP server updated to also
embed new content and upsert into `public.embeddings`.
## Open Items
- [ ] Final names (target database, schemas, MCP users) — picked at
execution time.
- [ ] pgvector status — Phase 0 must check `\dx` on each existing
database. If installed in OB1's database, no new install needed; if
absent, install during consolidation.
- [ ] Embedding model + dimension — inherited from whatever OB1 is using.
Phase 0 confirms.
- [ ] claude.ai chat capture path — no existing mechanism captures these.
Out of scope for initial reflection job inputs unless a capture path
lands first. If it doesn't, the job runs on Dispatch + Claude Code
streams only and chat captures get added later. **Flag for execution:
confirm with Travis whether to attempt a capture path in this plan or
explicitly defer.**
- [ ] Dispatch pre-compact hook timing — Phase 7 of the original ob1
deployment plan owns this. If it hasn't shipped, the reflection job's
Dispatch input is empty until it does. **Worth confirming current state
before starting Phase 4 of this plan.**
- [ ] Reflection prompt design — what does the synthesizer actually do?
Read N captures + relevant vault notes from the day → extract
generalizable lessons → produce structured output. The prompt itself
is real work; placeholder for now.
- [ ] Noise filtering for captures — what gets included in the daily
reflection window? All captures? Captures above some signal threshold?
Excludes routine command output? **Design step in Phase 4.**
- [ ] Reflection deduplication — re-running the job over overlapping
windows could produce near-duplicate lessons. Strategy: confidence-
weighted merge, manual review, or first-write-wins? **Design step in
Phase 4.**
## Phases
### Phase 0 — Inspection and prep
- [ ] Confirm Postgres instance hosts only `wiki`, `ob1`, `trellis`
(nothing else).
- [ ] `\dx` on each database — record which has pgvector installed.
- [ ] Inspect each database's schema: confirm table list, row counts,
any vector columns and their dimensions.
- [ ] Identify OB1's current embedding model (env var, code reference,
or configuration file).
- [ ] Inventory each MCP server's connection configuration: where
credentials live, how to update them.
- [ ] Confirm backup destination on the NAS is reachable over Tailscale
with sufficient space.
- [ ] Document current state in a session note before any changes.
### Phase 1 — Backups and rollback prep
- [ ] `pg_dumpall` of the full instance to NAS (full safety net).
- [ ] Per-database `pg_dump` of `wiki`, `ob1`, `trellis` to NAS
(granular restore targets).
- [ ] Verify dumps are readable (test restore to a scratch database).
- [ ] Write rollback runbook: how to restore from dumps and revert MCP
server connection strings if cutover fails.
### Phase 2 — Consolidation
- [ ] Pick final names: target database, schema names, MCP user names.
- [ ] If OB1's database is the target and is named appropriately, use
in place. Otherwise, rename or create new and restore OB1's content.
- [ ] Ensure pgvector is installed on the target database
(`CREATE EXTENSION IF NOT EXISTS vector`).
- [ ] Create `wiki` and `trellis` schemas in the target database.
- [ ] Move OB1's existing tables into the `ob1` schema if not already
namespaced (`ALTER TABLE ... SET SCHEMA ob1`).
- [ ] Restore `wiki` dump into the `wiki` schema using `pg_dump`'s
`--schema=wiki` retargeting or a sed pass over the dump SQL.
- [ ] Restore `trellis` dump into the `trellis` schema the same way.
- [ ] Create the three MCP service users.
- [ ] Grant ownership and permissions per the locked decisions.
- [ ] Set per-user `search_path` defaults.
- [ ] Verify row counts in each schema match pre-migration counts.
### Phase 3 — MCP server updates
- [ ] Update herby MCP server: new connection string (target DB,
`herby_mcp` user), confirm `search_path` resolution works without
qualifying table names.
- [ ] Update OB1's MCP server: same shape, `ob1_mcp` user.
- [ ] Update trellis MCP server: same shape, `trellis_mcp` user.
- [ ] Restart each MCP server, verify health, run one representative
tool from each (`list_vault_projects`, OB1 search, `list_threads`).
- [ ] **Positive isolation test:** confirm each MCP user is *rejected*
when attempting to write outside its owned schema. (E.g., `herby_mcp`
attempting `INSERT INTO ob1.captures` fails with permission error.)
- [ ] Update Lovebug's connection string if needed.
### Phase 4 — OB1 reflection layer (foundation)
- [ ] Create `ob1.reflections` table per the locked schema.
- [ ] Create `public.embeddings` table per the locked schema.
- [ ] Add the unique constraint on `(source_schema, source_table,
source_id, embedding_model)`.
- [ ] Add HNSW or IVFFlat index on `public.embeddings.embedding`
(model choice depends on row count and recall/speed trade — research
current pgvector defaults at execution time).
- [ ] Decide noise filtering rules for capture inclusion.
- [ ] Design the reflection prompt: input shape (captures from window
+ vault context), output shape (structured object matching
`ob1.reflections`).
- [ ] Decide dedup strategy.
### Phase 5 — Vault indexing
- [ ] Confirm embedding model + dimension from Phase 0.
- [ ] Write backfill script: walks all `wiki.notes`, embeds each
(body + title + tag context — decide chunking), inserts into
`public.embeddings`.
- [ ] Run backfill, verify row count matches `wiki.notes` count.
- [ ] Update herby MCP server's `save_vault_note`: embed new content
and upsert into `public.embeddings` on every write.
- [ ] Verify: write a new vault note via MCP, confirm embedding row
appears in `public.embeddings` referencing the new note.
### Phase 6 — Reflection job
- [ ] Implement the reflection job (Python script + cron entry, or
systemd timer).
- [ ] Job runs as `ob1_mcp` user; connects with its credentials.
- [ ] Inputs:
- Captures from the last 24 hours (or configured window)
- Relevant vault notes pulled via semantic search against the day's
captures
- [ ] Generates reflection objects via the chosen model.
- [ ] Writes to `ob1.reflections`, embeds each, upserts to
`public.embeddings`.
- [ ] Schedule for 3am local.
- [ ] First-run validation: run manually, inspect output, iterate on
prompt before letting cron take over.
### Phase 7 — End-to-end verification
- [ ] Semantic search query that hits all three content types in one
result set: vault notes, captures, reflections. Example query: "what
do we know about VLAN tagging" should return relevant rows from
multiple sources.
- [ ] Reflection citations resolve correctly — pick a recent reflection,
follow each citation back to its source row, confirm the join works.
- [ ] Permission model holds under stress — try writes from each MCP
user into other schemas; all should fail.
- [ ] Lovebug can do its job — confirm Lovebug's read patterns still
work under the new connection.
### Phase 8 — Cleanup
- [ ] Drop the old `wiki` and `trellis` databases (only after multi-day
verification period — keep dumps regardless).
- [ ] Archive pre-migration dumps to long-term backup location on NAS.
- [ ] Update connection-string documentation (KeePassXC, any internal
docs).
- [ ] Write a session note capturing the final shape, role model, and
operational commands (per-schema backup, restore, etc.).
## Notes
### Why consolidate at all
The three databases grew separately, not by deliberate isolation choice.
The reflection layer wants to cross all three constantly (cite vault notes
from captures from reflections). Cross-database queries in Postgres are
possible via FDW but every cross-DB query is a small tax — connection
management, transaction scope, FDW config. Schemas in one database make
cross-cutting queries free.
### Why isolation by user, not by database
You already separate concerns by *identity* across the stack — `travadmin`
vs `herbyadmin` on hosts, Authentik in front of MCP servers, GitHub
deploy keys scoped per machine. The pattern is consistent: separate the
*who*, not the *where*. Three MCP service users with schema-scoped
permissions matches that pattern. Postgres rejects writes outside an
MCP's owned schema at the database level — stronger than relying on the
MCP's code to behave.
### Why two layers in OB1
Same reason the wiki has two layers. Mixing raw captures and synthesized
lessons in one table pollutes semantic neighbors and degrades retrieval
quality as the reflection corpus grows. The wiki design explicitly
rejected this; OB1 deserves the same discipline.
### Why a shared embeddings table
Reflection retrieval is fundamentally cross-source. The shared table
makes "find similar to X across everything" the default query path.
Per-table vector columns would force `UNION ALL` queries across every
content type — workable but easy to forget a table when a new content
type is added later. Shared table is more extensible.
### Why vault indexing belongs in this plan
Without it, reflections can only cite captures and reflections — never
vault notes. The "pull anything on LLM memory projects" use case that
motivated this whole design needs vault content searchable. Building
reflection first and indexing later means the reflection layer is half-
useful for weeks.
### Why permissions are conservative now
Easier to grant access later than to revoke it after agents have come
to expect it. Starting with only OB1 having cross-schema reads (the
minimum needed for reflection to work) and revisiting after real usage
tells us what we actually need versus what we thought we needed.
### Risks
- **Cutover failure**: mitigated by full dumps and rehearsed rollback.
- **Embedding model mismatch**: if Phase 0 reveals OB1 is on a model
that's expensive to re-embed at scale, the choice of staying vs.
switching becomes load-bearing. Currently small corpus means low cost
either way; flag if discovered.
- **Reflection prompt quality**: bad prompts produce noisy reflections
that pollute retrieval. Phase 6's first-run validation gate is the
protection — don't let cron take over until manual runs produce good
output.
- **Permission boundary holes**: positive isolation tests in Phase 3 are
the protection. Verify rejection, not just success.
### What this plan does not do
- Does not redesign the trellis or wiki MCP servers beyond connection
changes.
- Does not implement claude.ai chat capture (open item).
- Does not address the Dispatch pre-compact hook (lives in the original
ob1 deployment plan, Phase 7).
- Does not tighten Lovebug's permissions (deferred; natural follow-on).
- Does not move Postgres off the current host or change its operational
footprint beyond schema reorganization.