adding 2 files
This commit is contained in:
Travis Herbranson
parent
659b329199
commit
52e3e76250
@ -0,0 +1,298 @@
|
|||||||
|
---
|
||||||
|
created: '2026-05-15'
|
||||||
|
path: Sources/Homelab
|
||||||
|
project: postgres-consolidation-reflection-layer
|
||||||
|
status: active
|
||||||
|
tags:
|
||||||
|
- ob1
|
||||||
|
- mcp
|
||||||
|
- pgvector
|
||||||
|
- embeddings
|
||||||
|
- session-note
|
||||||
|
type: session-note
|
||||||
|
updated: '2026-05-15'
|
||||||
|
worker: 2
|
||||||
|
---
|
||||||
|
|
||||||
|
# Postgres Consolidation — Worker 2, Phase 0 recon
|
||||||
|
|
||||||
|
Snapshot of the homelab Postgres instance before any consolidation
|
||||||
|
work. Recorded from `homelab-postgres` (pgvector/pgvector:pg17),
|
||||||
|
container in `/opt/projects/docker/postgres/`, port `127.0.0.1:5433`,
|
||||||
|
volume `ob1_data` (external).
|
||||||
|
|
||||||
|
## Postgres instance
|
||||||
|
|
||||||
|
- Container: `homelab-postgres` (image `pgvector/pgvector:pg17`)
|
||||||
|
- Volume: `ob1_data` (external — survives stack down)
|
||||||
|
- Network: `homelab` (alias `homelab-postgres`, also `postgres`)
|
||||||
|
- Healthy, up 40 hours.
|
||||||
|
- POSTGRES_DB default = `openbrain`.
|
||||||
|
|
||||||
|
## Databases on instance
|
||||||
|
|
||||||
|
| Name | Owner | Size | pgvector | Notes |
|
||||||
|
|-------------|-----------------|---------|----------|-------|
|
||||||
|
| `herbylab` | `herbylab_user` | 7958 kB | NO | Holds the herby-dev artifact store |
|
||||||
|
| `openbrain` | `postgres` | 8838 kB | YES (0.8.2) | OB1 captures live here |
|
||||||
|
| `trellis` | `trellis_user` | 8262 kB | NO | Trellis MCP threads/events |
|
||||||
|
| `postgres` | `postgres` | 7478 kB | - | default; not touched |
|
||||||
|
|
||||||
|
No other user databases. Plan's "instance hosts only wiki, ob1, trellis"
|
||||||
|
holds — under the actual names `herbylab` / `openbrain` / `trellis`.
|
||||||
|
|
||||||
|
## Existing roles
|
||||||
|
|
||||||
|
- `postgres` — Superuser, Create role, Create DB, Replication, Bypass RLS
|
||||||
|
- `herbylab_user` — owns `herbylab.herbylab` schema
|
||||||
|
- `trellis_user` — owns `trellis.trellis` schema
|
||||||
|
|
||||||
|
No `lovebug` role exists. No `ob1_mcp` / `vault_mcp` / `trellis_mcp`
|
||||||
|
service users yet — these come in Phase 2.
|
||||||
|
|
||||||
|
## Schemas, tables, row counts
|
||||||
|
|
||||||
|
### `herbylab` DB
|
||||||
|
- Schemas: `herbylab` (owner `herbylab_user`), `public`.
|
||||||
|
- Tables in `herbylab`:
|
||||||
|
- `projects` — 0 rows (8192 B; one empty heap page)
|
||||||
|
- `artifacts` — 0 rows (8192 B; one empty heap page)
|
||||||
|
- `alembic_version` — 1 row (`9c7e3f1a2b4d`)
|
||||||
|
- Custom enums: `herbylab.project_status`, `herbylab.artifact_type`, `herbylab.project_domain`.
|
||||||
|
- Triggers / functions: `projects_notify` (NOTIFY on row change), `artifacts_set_updated_at`.
|
||||||
|
- No vector columns.
|
||||||
|
|
||||||
|
**Why empty (verified, not a missed lookup):** the schema was created
|
||||||
|
by alembic migration `9c7e3f1a2b4d` on the unmerged branch
|
||||||
|
`claude/competent-knuth-de9483` (commit `90bc6a9 Phase 1: herbylab
|
||||||
|
Postgres schema infrastructure`). That branch also has commits
|
||||||
|
`8e408c5 Phase 2: polymorphic artifact tools backed by Postgres +
|
||||||
|
vault` and `78ec094 Phase 3: Trello projection via n8n` which would
|
||||||
|
have wired writes into the schema, but **the branch is not merged to
|
||||||
|
`main`.**
|
||||||
|
|
||||||
|
The deployed `herbydev-mcp` runs `main`'s code, which has zero DB
|
||||||
|
access:
|
||||||
|
- `pyproject.toml` declares no DB driver (no psycopg / sqlalchemy /
|
||||||
|
asyncpg).
|
||||||
|
- `mcp_server/tools/memory.py` and `mcp_server/tools/tasks.py` are
|
||||||
|
empty stub files marked "Phase 2. Do not implement until workflow
|
||||||
|
patterns are established."
|
||||||
|
- `mcp_server/tools/vault.py` writes via `core/git_writer.py` →
|
||||||
|
filesystem + git commit + push to gitea. No database insertion.
|
||||||
|
- `HERBYLAB_DATABASE_URL` is plumbed through compose env but never
|
||||||
|
read by the running code.
|
||||||
|
|
||||||
|
So the DB schema exists from an integration-test run of the unmerged
|
||||||
|
branch's migration; no code path on `main` ever populates it.
|
||||||
|
|
||||||
|
Implication for consolidation: the `wiki` schema in `petalbrain` is
|
||||||
|
genuinely empty content-wise. Worker 4's vault-indexing source is the
|
||||||
|
filesystem (`/opt/projects/wiki-vault/`), not a `wiki.notes` /
|
||||||
|
`wiki.artifacts` table. If `competent-knuth-de9483` lands later, its
|
||||||
|
artifact tooling will need to be re-pointed at `petalbrain.wiki.*`
|
||||||
|
instead of `herbylab.herbylab.*`, and the Phase 4 reflection design
|
||||||
|
should account for `wiki.artifacts` becoming a populated source over
|
||||||
|
time.
|
||||||
|
|
||||||
|
### `openbrain` DB
|
||||||
|
- Schemas: `public` only.
|
||||||
|
- Tables in `public`:
|
||||||
|
- `thoughts` — 70 rows (live; daemons writing today)
|
||||||
|
- pgvector 0.8.2 in `public`.
|
||||||
|
|
||||||
|
`public.thoughts` shape (relevant columns):
|
||||||
|
|
||||||
|
```
|
||||||
|
id bigint PK (nextval thoughts_id_seq)
|
||||||
|
content text NOT NULL
|
||||||
|
embedding vector(768)
|
||||||
|
metadata jsonb DEFAULT '{}'
|
||||||
|
user_id text NOT NULL DEFAULT 'travis'
|
||||||
|
created_at timestamptz DEFAULT CURRENT_TIMESTAMP
|
||||||
|
updated_at timestamptz DEFAULT CURRENT_TIMESTAMP
|
||||||
|
```
|
||||||
|
|
||||||
|
Existing indexes on `public.thoughts`:
|
||||||
|
- `thoughts_pkey` (id)
|
||||||
|
- `idx_thoughts_created_at` (created_at DESC)
|
||||||
|
- `idx_thoughts_embedding_hnsw` — **hnsw (embedding vector_cosine_ops) WITH (m=16, ef_construction=64)** — already matches the locked Phase 4 spec for `public.embeddings`.
|
||||||
|
- `idx_thoughts_fingerprint` — UNIQUE (md5(content), user_id)
|
||||||
|
- `idx_thoughts_metadata` — gin (metadata)
|
||||||
|
- `idx_thoughts_user_created` (user_id, created_at DESC)
|
||||||
|
- `idx_thoughts_user_id` (user_id)
|
||||||
|
|
||||||
|
Additional objects:
|
||||||
|
- Function `match_thoughts(query_embedding vector, match_threshold float, match_count int, filter jsonb, p_user_id text)` — bare-`vector` param (no dim), so dimension changes flow through unchanged.
|
||||||
|
- Trigger `thoughts_updated_at` -> `update_updated_at()`.
|
||||||
|
|
||||||
|
Live writer state: 70 rows total, 0 with NULL embedding, 2 still flagged `metadata->>'pending' = 'true'` (enricher backlog). Most recent row id 255 at `2026-05-16 02:07:23 UTC`. Watcher and enricher daemons confirmed live.
|
||||||
|
|
||||||
|
### `trellis` DB
|
||||||
|
- Schemas: `trellis` (owner `trellis_user`), `public`.
|
||||||
|
- Tables in `trellis`:
|
||||||
|
- `threads` — 13 rows
|
||||||
|
- `events` — 50 rows
|
||||||
|
- `tags` — 42 rows
|
||||||
|
- `thread_tags` — 55 rows
|
||||||
|
- `alembic_version` — 1 row
|
||||||
|
- No vector columns.
|
||||||
|
|
||||||
|
## OB1 embedding model
|
||||||
|
|
||||||
|
From `ob1-mcp` container env (and `/opt/projects/homelab/ob1-deploy/compose.yml`):
|
||||||
|
|
||||||
|
- `EMBEDDING_PROVIDER=ollama`
|
||||||
|
- `OLLAMA_URL=http://ollama:11434`
|
||||||
|
- `EMBEDDING_MODEL=nomic-embed-text` (768-d output)
|
||||||
|
- `CHAT_MODEL=openai/gpt-4o-mini` (mcp side)
|
||||||
|
- `CHAT_MODEL=llama3.2:3b` (enricher side)
|
||||||
|
|
||||||
|
Migration trail:
|
||||||
|
- `init.sql` originally declared `vector(1536)` for `text-embedding-3-small`.
|
||||||
|
- `migrations/001_embedding_dim_768.sql` (applied 2026-05-14) dropped the
|
||||||
|
HNSW index, altered the column to `vector(768)`, recreated the index.
|
||||||
|
Required `thoughts` to be empty pre-migration.
|
||||||
|
|
||||||
|
**Conclusion:** the shared `public.embeddings` table goes in as
|
||||||
|
`vector(768)` to match `nomic-embed-text`, no model switch needed. HNSW
|
||||||
|
index spec from Travis's overrides (`vector_cosine_ops, m=16,
|
||||||
|
ef_construction=64`) matches the existing OB1 convention exactly.
|
||||||
|
|
||||||
|
## MCP server connection configurations
|
||||||
|
|
||||||
|
| Container | Image | DB host / DB / user | Live? |
|
||||||
|
|-----------------|--------------------|---------------------|-------|
|
||||||
|
| `herbydev-mcp` | `herbydev-mcp` | `homelab-postgres:5432/herbylab` as `herbylab_user` | Up 40h |
|
||||||
|
| `vault-mcp` | `vault-mcp` | **`vault-postgres:5432/herbylab` as `herbylab_user`** | Up 59m |
|
||||||
|
| `ob1-mcp` | `ob1-mcp` | `homelab-postgres:5432/openbrain` as `postgres` | Up 14h |
|
||||||
|
| `ob1-watcher` | `ob1-watcher` | (talks to ob1-mcp HTTP, not DB directly) | Up 26h |
|
||||||
|
| `ob1-enricher` | `ob1-enricher` | (uses `OLLAMA_URL`; DB password set but no direct connect string visible) | Up 14h |
|
||||||
|
| `trellis-mcp` | `trellis-trellis-mcp` | `homelab-postgres:5432/trellis` as `trellis_user` | Up 40h healthy |
|
||||||
|
|
||||||
|
Connection strings live in:
|
||||||
|
- `vault-mcp` / `herbydev-mcp` — `/opt/projects/homelab/herby-dev/deploy/.env` (`HERBYLAB_DATABASE_URL`)
|
||||||
|
- `ob1-mcp` / `ob1-enricher` — `/opt/projects/homelab/ob1-deploy/.env` (`POSTGRES_PASSWORD` + per-service `DB_*` env in `compose.yml`)
|
||||||
|
- `trellis-mcp` — `/opt/projects/homelab/trellis-mcp/` (compose env, `DATABASE_URL`)
|
||||||
|
|
||||||
|
**Anomaly:** `vault-mcp` references hostname `vault-postgres` which
|
||||||
|
does not resolve on the `homelab` network (alias is `homelab-postgres`
|
||||||
|
+ `postgres`). This is consistent with Travis's mid-flight rename
|
||||||
|
rollout — the container is currently running OLD code per the
|
||||||
|
priming, and the connection-string update is part of the unfinished
|
||||||
|
rollout. Not blocking Phase 1, but Phase 3 must coordinate with the
|
||||||
|
rename completion.
|
||||||
|
|
||||||
|
## Backup destination
|
||||||
|
|
||||||
|
**Local-only on herbydev** (Travis decision, 2026-05-15). The NAS
|
||||||
|
reference in the plan was unintended — disregard it.
|
||||||
|
|
||||||
|
- Destination directory: `/opt/backups/postgres-consolidation/`
|
||||||
|
(does not exist yet; Phase 1 step zero is `mkdir -p` it).
|
||||||
|
- Local disk on `/`: 14 G free of 97 G (86% used).
|
||||||
|
- Combined DB size is ~25 MB, so headroom is ample for both Phase 1
|
||||||
|
bulk dumps and Phase 2-start fresh dumps.
|
||||||
|
- Format: `pg_dumpall` (full instance) + per-database `pg_dump` in
|
||||||
|
custom format (`-Fc`), gzipped. Filenames include UTC timestamp.
|
||||||
|
- No NAS / Tailscale / off-host destination involved.
|
||||||
|
|
||||||
|
## Coordination state — running daemons
|
||||||
|
|
||||||
|
`ob1-watcher` and `ob1-enricher` are both live and writing to
|
||||||
|
`openbrain.public.thoughts` (latest write ~5 minutes before this
|
||||||
|
note). Per priming, the Phase 2 `ALTER TABLE public.thoughts SET
|
||||||
|
SCHEMA ob1` step is the cutover gate: daemons must be paused, table
|
||||||
|
moved + ob1.thoughts shape rewrite by Worker 3, then daemons resumed.
|
||||||
|
Worker 2 stops at the announcement of readiness for that move and
|
||||||
|
waits.
|
||||||
|
|
||||||
|
## Naming reconciliation (plan vs. reality)
|
||||||
|
|
||||||
|
Plan vocabulary `wiki / ob1 / trellis` will land as the **schema**
|
||||||
|
names in the consolidated database, per Travis's locked decisions.
|
||||||
|
Source DB → target schema mapping:
|
||||||
|
|
||||||
|
- `herbylab` DB content → `wiki` schema (owned by `vault_mcp` per
|
||||||
|
override; was `herby_mcp` in the plan)
|
||||||
|
- `openbrain` DB content → `ob1` schema; `public.thoughts` becomes
|
||||||
|
`ob1.thoughts` (preserving `thoughts` vocabulary per override; plan
|
||||||
|
said `ob1.captures`)
|
||||||
|
- `trellis` DB content → `trellis` schema (owned by `trellis_mcp`)
|
||||||
|
|
||||||
|
Plus shared `public.embeddings` per Phase 4.
|
||||||
|
|
||||||
|
**Open: target database name itself.** OB1's database is currently
|
||||||
|
`openbrain`. The plan says "if OB1's database is the target and is
|
||||||
|
named appropriately, use in place. Otherwise, rename or create new."
|
||||||
|
`openbrain` becomes a misleading name once it also holds wiki +
|
||||||
|
trellis schemas. Worth a deliberate rename (e.g., `homelab`,
|
||||||
|
`petalbrain`, or similar) before Phase 2 work begins.
|
||||||
|
|
||||||
|
## Phase 1 readiness assessment
|
||||||
|
|
||||||
|
Ready:
|
||||||
|
- Source DB sizes are trivial; dumps will be small and fast.
|
||||||
|
- pgvector already present in target DB (`openbrain`); no extension
|
||||||
|
install needed.
|
||||||
|
- Embedding model + dim are confirmed and stable (`nomic-embed-text` /
|
||||||
|
768) — no re-embed forced.
|
||||||
|
- Existing HNSW spec on `public.thoughts` matches the locked
|
||||||
|
`public.embeddings` HNSW spec exactly.
|
||||||
|
|
||||||
|
Not ready / needs decision:
|
||||||
|
- ~~NAS backup destination~~ **Resolved 2026-05-15**: local-only at
|
||||||
|
`/opt/backups/postgres-consolidation/`. No NAS involved.
|
||||||
|
- ~~Target database name~~ **Resolved 2026-05-15**: `petalbrain`.
|
||||||
|
`openbrain` renamed before Phase 2 destructive ops begin.
|
||||||
|
- **`wiki` schema content scope** — `herbylab` DB schema came from the
|
||||||
|
unmerged `claude/competent-knuth-de9483` branch's migration
|
||||||
|
`9c7e3f1a2b4d`; `main`'s deployed MCP doesn't open a DB connection
|
||||||
|
at all. Wiki content is filesystem-only under
|
||||||
|
`/opt/projects/wiki-vault/`. Worker 4's vault-indexing source must
|
||||||
|
walk the filesystem and write into `public.embeddings` directly.
|
||||||
|
See the herbylab-DB section above for the full evidence trail.
|
||||||
|
- **Connection-string anomaly in `vault-mcp`** (`vault-postgres`
|
||||||
|
hostname). Tied to Travis's rename rollout, blocks Phase 3 cutover
|
||||||
|
for that MCP only. Phase 1/2 unaffected.
|
||||||
|
|
||||||
|
Conservative call: gate before Phase 1 until the NAS destination and
|
||||||
|
target DB name decisions are made.
|
||||||
|
|
||||||
|
## Addendum — Phase 2-start fresh backup (Travis directive, 2026-05-15)
|
||||||
|
|
||||||
|
**In addition to Phase 1 bulk backups**, Phase 2 must begin with a
|
||||||
|
fresh `pg_dump` of each source database immediately before the first
|
||||||
|
destructive operation (no `CREATE SCHEMA`, no `ALTER TABLE`, no
|
||||||
|
restore — nothing destructive precedes this step).
|
||||||
|
|
||||||
|
Rationale: `ob1-watcher` is writing to `public.thoughts` continuously
|
||||||
|
(latest write ~5 min before this note). Time elapses between Phase 1
|
||||||
|
dumps and Phase 2 cutover, especially with gating. Any rows captured
|
||||||
|
in that window would be lost on rollback if Phase 1 dumps are the
|
||||||
|
only restore point.
|
||||||
|
|
||||||
|
Specifics:
|
||||||
|
- Run fresh `pg_dump` on `openbrain` first — it's the live writer and
|
||||||
|
the most critical. Then `herbylab` and `trellis` for symmetry.
|
||||||
|
- Destination: `/opt/backups/postgres-consolidation/` (same as Phase
|
||||||
|
1), gzipped custom format.
|
||||||
|
- Filename suffix: `-phase2-start-<UTC-timestamp>` to distinguish from
|
||||||
|
Phase 1 backups.
|
||||||
|
- Verify each fresh dump is readable via test restore to a scratch
|
||||||
|
database (same pattern as Phase 1).
|
||||||
|
- Only after the three fresh dumps are verified does Phase 2's first
|
||||||
|
destructive operation run.
|
||||||
|
|
||||||
|
This is not a replacement for Phase 1 dumps — Phase 1 remains the
|
||||||
|
broad safety net (full instance `pg_dumpall` + per-DB granular
|
||||||
|
dumps). Phase 2-start dumps are the just-before-disruption snapshot
|
||||||
|
that captures any in-flight writer activity.
|
||||||
|
|
||||||
|
Operational note: this is also the natural point to pause
|
||||||
|
`ob1-watcher` and `ob1-enricher`. Sequence:
|
||||||
|
1. Pause both daemons (`docker compose stop ob1-watcher ob1-enricher`).
|
||||||
|
2. Confirm no in-flight writes (`pg_stat_activity` quiet on `openbrain`).
|
||||||
|
3. Take the fresh per-DB dumps + verify.
|
||||||
|
4. Begin Phase 2 destructive work.
|
||||||
|
5. (Worker 3 resumes daemons after `ob1.thoughts` cutover lands.)
|
||||||
@ -0,0 +1,252 @@
|
|||||||
|
---
|
||||||
|
created: '2026-05-15'
|
||||||
|
path: Sources/Homelab
|
||||||
|
project: postgres-consolidation-reflection-layer
|
||||||
|
status: active
|
||||||
|
tags:
|
||||||
|
- ob1
|
||||||
|
- mcp
|
||||||
|
- pgvector
|
||||||
|
- embeddings
|
||||||
|
- session-note
|
||||||
|
type: session-note
|
||||||
|
updated: '2026-05-15'
|
||||||
|
worker: 2
|
||||||
|
phase: 2a
|
||||||
|
---
|
||||||
|
|
||||||
|
# Postgres Consolidation — Worker 2, Phase 2a result
|
||||||
|
|
||||||
|
Light Phase 2a complete. Disruption-free: no consumer connections
|
||||||
|
broken, no live writers paused, no schema rename. `openbrain` is still
|
||||||
|
the live DB name; `petalbrain` rename is Worker 3's cutover.
|
||||||
|
|
||||||
|
## What landed
|
||||||
|
|
||||||
|
Inside `openbrain` (the existing DB):
|
||||||
|
|
||||||
|
### New roles (no grants on `ob1` yet — that schema doesn't exist)
|
||||||
|
|
||||||
|
| Role | Schema-level grants | search_path |
|
||||||
|
|---------------|----------------------------------------------------------------------------|--------------------------------|
|
||||||
|
| `vault_mcp` | OWNER of `wiki` | `wiki, public` |
|
||||||
|
| `trellis_mcp` | OWNER of `trellis` | `trellis, public` |
|
||||||
|
| `ob1_mcp` | USAGE + SELECT on `wiki` and `trellis` | `ob1, public` (ob1 not yet created) |
|
||||||
|
| `lovebug` | USAGE + SELECT + INSERT/UPDATE/DELETE on `wiki` and `trellis` | `wiki, ob1, trellis, public` |
|
||||||
|
|
||||||
|
All four: LOGIN, NOSUPERUSER, NOCREATEDB, NOCREATEROLE, NOREPLICATION,
|
||||||
|
NOBYPASSRLS, INHERIT.
|
||||||
|
|
||||||
|
Default privileges configured so future tables owned by `vault_mcp`
|
||||||
|
(in `wiki`) and `trellis_mcp` (in `trellis`) inherit the same grant
|
||||||
|
model automatically.
|
||||||
|
|
||||||
|
Credentials at `/opt/backups/postgres-consolidation/credentials.env`
|
||||||
|
(mode 0600, herbyadmin-only). 64-char hex passwords. For KeePassXC
|
||||||
|
transcription.
|
||||||
|
|
||||||
|
### Schemas in `openbrain`
|
||||||
|
|
||||||
|
- `wiki` — restored from Phase 1 `herbylab-pg_dump`, then
|
||||||
|
`ALTER SCHEMA herbylab RENAME TO wiki`. Contains:
|
||||||
|
- tables `alembic_version` (1 row), `artifacts` (0 rows),
|
||||||
|
`projects` (0 rows)
|
||||||
|
- sequences `artifacts_id_seq`, `projects_id_seq` (linked to
|
||||||
|
IDENTITY columns)
|
||||||
|
- enums `artifact_type`, `project_status`, `project_domain`
|
||||||
|
- functions `artifacts_set_updated_at`, `notify_project_change`
|
||||||
|
- triggers `artifacts_updated_at_bump`, `projects_notify`
|
||||||
|
- FK constraint `projects.artifact_id` → `artifacts.id`
|
||||||
|
- all OID references updated transparently across the rename
|
||||||
|
|
||||||
|
- `trellis` — restored from Phase 1 `trellis-pg_dump` (same name in
|
||||||
|
source and target, no rename). Contains:
|
||||||
|
- tables `alembic_version` (1), `events` (50), `tags` (42),
|
||||||
|
`thread_tags` (55), `threads` (13)
|
||||||
|
- sequences `events_id_seq`, `tags_id_seq`, `threads_id_seq`
|
||||||
|
(OWNED BY linked)
|
||||||
|
- FK chain across `events ⇒ threads`, `thread_tags ⇒ threads`,
|
||||||
|
`thread_tags ⇒ tags`, `threads.parent_id ⇒ threads`
|
||||||
|
|
||||||
|
- `public` — **untouched**. `public.thoughts` still owned by
|
||||||
|
`postgres`, still receiving writes from `ob1-watcher` (75 rows at
|
||||||
|
end of Phase 2a, vs 74 at Phase 1 dump start, vs 70 at Phase 0
|
||||||
|
recon — `ob1-watcher` is healthy). Worker 3 moves this to `ob1`.
|
||||||
|
|
||||||
|
- `public.embeddings` — **new, empty**. Shared cross-schema embedding
|
||||||
|
store per locked spec:
|
||||||
|
|
||||||
|
```
|
||||||
|
public.embeddings (
|
||||||
|
id BIGSERIAL PRIMARY KEY,
|
||||||
|
source_schema TEXT NOT NULL,
|
||||||
|
source_table TEXT NOT NULL,
|
||||||
|
source_id BIGINT NOT NULL,
|
||||||
|
embedding VECTOR(768) NOT NULL,
|
||||||
|
embedding_model TEXT NOT NULL,
|
||||||
|
embedded_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||||
|
UNIQUE (source_schema, source_table, source_id, embedding_model)
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
Indexes:
|
||||||
|
- `embeddings_pkey` (id)
|
||||||
|
- `embeddings_source_uniq` UNIQUE (source_schema, source_table,
|
||||||
|
source_id, embedding_model)
|
||||||
|
- `idx_embeddings_hnsw` HNSW (embedding vector_cosine_ops) WITH
|
||||||
|
(m=16, ef_construction=64) — matches the existing
|
||||||
|
`idx_thoughts_embedding_hnsw` convention exactly
|
||||||
|
- `idx_embeddings_source` (source_schema, source_table, source_id)
|
||||||
|
— for orphan cleanup + source-aware queries
|
||||||
|
- `idx_embeddings_model` (embedding_model) — for the multi-model
|
||||||
|
coexistence case the plan calls out
|
||||||
|
|
||||||
|
Owned by `postgres` (neutral shared infrastructure). All four
|
||||||
|
service users have SELECT+INSERT+UPDATE+DELETE on the table and
|
||||||
|
SELECT+USAGE on the `embeddings_id_seq` sequence. No FK — orphan
|
||||||
|
cleanup is a separate periodic job (out of scope here per plan).
|
||||||
|
|
||||||
|
## What Phase 2a did NOT do
|
||||||
|
|
||||||
|
- Did not rename `openbrain` → `petalbrain`. That's a destructive
|
||||||
|
cutover (live consumers must disconnect for `ALTER DATABASE
|
||||||
|
RENAME`). Deferred to Worker 3.
|
||||||
|
- Did not create the `ob1` schema. Worker 3 owns the
|
||||||
|
`ALTER TABLE public.thoughts SET SCHEMA ob1` step + accompanying
|
||||||
|
shape rewrite.
|
||||||
|
- Did not touch `public.thoughts` in any way. Still in `public`,
|
||||||
|
still writable, still being written to.
|
||||||
|
- Did not pause `ob1-watcher` or `ob1-enricher`. Both still live.
|
||||||
|
- Did not modify any MCP container connection strings or env files.
|
||||||
|
All MCPs still connect to `openbrain` as before.
|
||||||
|
- Did not touch the source `herbylab` or `trellis` databases. Both
|
||||||
|
still exist on the instance with their original content; cleanup
|
||||||
|
is Phase 8 of the plan.
|
||||||
|
|
||||||
|
## Verification done
|
||||||
|
|
||||||
|
### Row counts in openbrain match dump-source counts exactly
|
||||||
|
|
||||||
|
| Schema/Table | Live count | Dump-source count |
|
||||||
|
|---------------------------|-----------:|------------------:|
|
||||||
|
| `wiki.projects` | 0 | 0 (herbylab) |
|
||||||
|
| `wiki.artifacts` | 0 | 0 (herbylab) |
|
||||||
|
| `wiki.alembic_version` | 1 | 1 (herbylab) |
|
||||||
|
| `trellis.threads` | 13 | 13 |
|
||||||
|
| `trellis.events` | 50 | 50 |
|
||||||
|
| `trellis.tags` | 42 | 42 |
|
||||||
|
| `trellis.thread_tags` | 55 | 55 |
|
||||||
|
| `trellis.alembic_version` | 1 | 1 |
|
||||||
|
| `public.thoughts` | 75 | (live; was 74 at dump time) |
|
||||||
|
| `public.embeddings` | 0 | (new) |
|
||||||
|
|
||||||
|
Source `herbylab` DB and source `trellis` DB confirmed unchanged.
|
||||||
|
|
||||||
|
### Permission boundary smoke tests (positive isolation per plan Phase 3)
|
||||||
|
|
||||||
|
All ran via direct postgres connection on
|
||||||
|
`postgresql://<role>:<pw>@localhost:5432/openbrain`:
|
||||||
|
|
||||||
|
| Test | Expected | Result |
|
||||||
|
|--------------------------------------------------|----------|--------|
|
||||||
|
| `vault_mcp` SELECT `wiki.artifacts` | succeed | OK |
|
||||||
|
| `vault_mcp` INSERT `trellis.threads` | DENY | `ERROR: permission denied for schema trellis` |
|
||||||
|
| `trellis_mcp` SELECT `wiki.artifacts` | DENY | `ERROR: permission denied for schema wiki` |
|
||||||
|
| `ob1_mcp` SELECT `wiki.artifacts` + `trellis.threads` | succeed | OK (counts 0 + 13) |
|
||||||
|
| `ob1_mcp` INSERT `wiki.artifacts` | DENY | `ERROR: permission denied for table artifacts` |
|
||||||
|
| `vault_mcp` INSERT `public.embeddings` | succeed | OK (id 1) |
|
||||||
|
| `ob1_mcp` INSERT `public.embeddings` | succeed | OK (id 2) |
|
||||||
|
| `trellis_mcp` INSERT `public.embeddings` | succeed | OK (id 3) |
|
||||||
|
| `lovebug` INSERT `public.embeddings` | succeed | OK (id 4) |
|
||||||
|
|
||||||
|
Smoke-test rows (4) deleted after verification;
|
||||||
|
`public.embeddings` back to 0 rows.
|
||||||
|
|
||||||
|
## Surprises / notes
|
||||||
|
|
||||||
|
- **Sequence ownership followed table.** First pass on `ALTER OWNER`
|
||||||
|
failed with `cannot change owner of sequence "artifacts_id_seq" —
|
||||||
|
Sequence "artifacts_id_seq" is linked to table "artifacts"`. The
|
||||||
|
herbylab dump uses `GENERATED ALWAYS AS IDENTITY`, whose sequence
|
||||||
|
is owned by the table and cannot be reassigned independently.
|
||||||
|
Fixed by excluding `relkind='S'` from the table loop; sequence
|
||||||
|
ownership now follows the table automatically. Trellis sequences
|
||||||
|
use SERIAL/OWNED BY which has the same behavior. Caught + handled
|
||||||
|
in one retry; final state correct.
|
||||||
|
- **`ob1_mcp.search_path = ob1, public`** is set now even though the
|
||||||
|
`ob1` schema doesn't exist yet. Once Worker 3 creates the schema
|
||||||
|
in the same DB (whatever its final name), unqualified table refs
|
||||||
|
from `ob1-mcp`'s connection will resolve there automatically.
|
||||||
|
- **No `vault-mcp` env update** was needed for Phase 2a. The current
|
||||||
|
`HERBYLAB_DATABASE_URL` in `vault-mcp` points at hostname
|
||||||
|
`vault-postgres` (still anomalous — part of Travis's rename
|
||||||
|
rollout) and DB `herbylab`. Neither is wired into the consolidated
|
||||||
|
state yet. Phase 3 of the original plan handles MCP connection
|
||||||
|
cutover.
|
||||||
|
|
||||||
|
## Worker 3 readiness assessment
|
||||||
|
|
||||||
|
Worker 3's cutover window is the next destructive step. From the
|
||||||
|
priming, Worker 3 must:
|
||||||
|
|
||||||
|
1. Pause `ob1-watcher` + `ob1-enricher` + `ob1-mcp`.
|
||||||
|
2. Take Phase 2-start fresh `pg_dump` of `openbrain`, `herbylab`,
|
||||||
|
`trellis` to `/opt/backups/postgres-consolidation/` with suffix
|
||||||
|
`-phase2-start-<UTC-timestamp>`. Verify via test restore.
|
||||||
|
3. `ALTER DATABASE openbrain RENAME TO petalbrain` (now safe — no
|
||||||
|
live connections).
|
||||||
|
4. Connect to `petalbrain` and `ALTER TABLE public.thoughts SET
|
||||||
|
SCHEMA ob1`, plus all dependent objects (indexes are intra-table
|
||||||
|
so they follow; the `match_thoughts` function references
|
||||||
|
`thoughts` directly without schema qualification so search_path
|
||||||
|
must include `ob1` when ob1_mcp calls it — already set; the
|
||||||
|
`thoughts_updated_at` trigger follows the table; `update_updated_at()`
|
||||||
|
function lives in `public` and should be moved or be referenced
|
||||||
|
cross-schema).
|
||||||
|
5. Create `ob1` schema if `SET SCHEMA` didn't auto-create it
|
||||||
|
(`SET SCHEMA` does create the schema if it didn't exist — but
|
||||||
|
actually it requires the schema to pre-exist; Worker 3 should
|
||||||
|
`CREATE SCHEMA ob1 AUTHORIZATION ob1_mcp` first).
|
||||||
|
6. ob1-specific grants: `ob1_mcp` owns `ob1.*`; `lovebug` gets full
|
||||||
|
RW; `vault_mcp` and `trellis_mcp` get nothing.
|
||||||
|
7. Backfill embeddings: for each of the 75-ish rows in
|
||||||
|
`ob1.thoughts`, insert into `public.embeddings` with
|
||||||
|
`source_schema='ob1'`, `source_table='thoughts'`, `source_id=<row
|
||||||
|
id>`, `embedding=<existing embedding column value>`,
|
||||||
|
`embedding_model='nomic-embed-text'`.
|
||||||
|
8. Decide: keep the `ob1.thoughts.embedding` column post-backfill
|
||||||
|
(redundant with `public.embeddings`) or drop it. Plan says all
|
||||||
|
embeddings live in `public.embeddings`; suggests dropping. But
|
||||||
|
that changes the row shape for `ob1-mcp`'s code. Worker 3 call.
|
||||||
|
9. Rewrite `ob1-mcp`'s `add_thought` and `search_thoughts` for the
|
||||||
|
new shape (target table `ob1.thoughts`, embedding pushed to
|
||||||
|
`public.embeddings` separately, search joins).
|
||||||
|
10. Update connection strings on `ob1-mcp`, `ob1-watcher`,
|
||||||
|
`ob1-enricher` from DB `openbrain` to DB `petalbrain`. Watcher
|
||||||
|
talks to ob1-mcp HTTP only, so its env may not need DB host
|
||||||
|
change — verify.
|
||||||
|
11. Update `trellis-mcp` and `herbydev-mcp` / `vault-mcp` connection
|
||||||
|
strings from their current DBs (`trellis`, `herbylab`) to the
|
||||||
|
consolidated `petalbrain` with their service-user credentials
|
||||||
|
from `credentials.env`.
|
||||||
|
12. Resume `ob1-watcher` + `ob1-enricher`.
|
||||||
|
13. Verify: row counts, semantic search query roundtrip, NOTIFY
|
||||||
|
trigger on `wiki.projects` still emits.
|
||||||
|
|
||||||
|
Phase 2a outputs Worker 3 will need:
|
||||||
|
- Service user passwords in
|
||||||
|
`/opt/backups/postgres-consolidation/credentials.env`.
|
||||||
|
- Phase 1 backup set in `/opt/backups/postgres-consolidation/` plus
|
||||||
|
the rollback runbook there.
|
||||||
|
- This session note.
|
||||||
|
|
||||||
|
## Holding pattern
|
||||||
|
|
||||||
|
`ob1-watcher` and `ob1-enricher` continue writing to
|
||||||
|
`public.thoughts`. Each minute that passes, the gap between the Phase
|
||||||
|
1 dump's `thoughts` snapshot (74) and the cutover-time count grows.
|
||||||
|
Phase 2-start fresh dump shrinks that gap to seconds when Worker 3
|
||||||
|
runs. Until then: no degradation, no risk; just normal capture.
|
||||||
|
|
||||||
|
Worker 2 standing by. No further actions until Worker 3 spawns or
|
||||||
|
Travis redirects.
|
||||||
Loading…
Reference in New Issue
Block a user