diff --git a/Sources/Homelab/vault-semantic-search.md b/Sources/Homelab/vault-semantic-search.md new file mode 100644 index 0000000..c752c39 --- /dev/null +++ b/Sources/Homelab/vault-semantic-search.md @@ -0,0 +1,226 @@ +--- +created: '2026-05-17' +path: Sources/Homelab +project: vault-semantic-search +tags: +- mcp +- pgvector +- embeddings +- wiki +type: project-plan +--- + +# Vault Semantic Search + +## Goal + +Add a `vault:search` MCP tool that takes a query string (plus optional +metadata filters) and returns vault notes ranked by semantic similarity. +The tool runs entirely against the `wiki` schema and reuses the embedding +infrastructure built by the Postgres consolidation plan. + +Concrete use case: "search the vault for context" returns relevant notes +without needing to remember slugs or tag names. Semantic search replaces +the current "list everything, filter client-side" pattern. + +This plan is scoped to vault search only. A future unified memory search +spanning vault + thoughts + reflections is anticipated via a `scope` +parameter, but is deferred until vault search is in use and shaping +requirements are clearer. + +## Locked Decisions + +### Tool surface + +- Name: `vault:search` +- Read-only. No writes, no side effects. +- Single MCP tool — no separate "by tag" or "by date" variants. All + filtering is parameterized on this one tool. + +### Inputs + +- `query` (required, string) — natural-language search query +- `tags` (optional, list of strings) — restrict to notes carrying any of + these tags +- `domain` (optional, string) — one of the schema's domain values + (homelab, dev, venture, reference) +- `status` (optional, list of strings) — one or more lifecycle statuses +- `note_type` (optional, string) — restrict to project-plan, + session-notes, entity, topic-landing, or synthesis +- `date_range` (optional, object with `start` and `end`) — filter by + `updated_at` timestamp range +- `limit` (optional, int, default 10) — max results to return + +Filters compose via AND. Vector similarity ranks within the filtered +candidate set. + +### Output shape (minimal) + +``` +[ + { + "slug": "ob1-deployment", + "title": "OB1 Main Deployment", + "path": "Sources/Homelab/ob1-deployment.md", + "score": 0.82 + }, + ... +] +``` + +Score is the cosine similarity (or pgvector distance converted to +similarity — implementation detail). Caller uses `vault:get_project` +to fetch full content for any result they want to read. + +Keeping content fetch separate from search keeps the search response +cheap and lets callers decide when full body content is worth the round +trip. + +### Embedding behavior + +- Query embedding runs **server-side** inside the vault MCP server. +- The server uses the **same embedding model** that produced the stored + vectors in `public.embeddings` (recorded in the `embedding_model` + column). +- Mismatches are not silently tolerated — if the configured query model + doesn't match the model recorded on the rows being searched, the + search returns an error rather than incorrect rankings. +- Embedding model selection is inherited from the consolidation plan; + this plan doesn't pick or change models. + +### Scope + +- Searches the `wiki` schema only. Both Sources and Wiki layers are + searchable; filter via `note_type` if a caller wants Sources-only or + Wiki-only. +- Does not search `ob1.thoughts` or `ob1.reflections`. That's a separate + tool (or this tool extended with a `scope` parameter) in a future plan. + +### Backend mechanics + +- Vector similarity query against `public.embeddings` filtered by + `source_schema = 'wiki'`. +- Filters applied via WHERE clauses joining `public.embeddings` to + `wiki.notes` (or whatever the canonical wiki content table is named). +- HNSW index on `public.embeddings.embedding` is the consolidation + plan's responsibility; this plan reuses it. + +## Open Items + +- [ ] Final SQL query shape — JOIN pattern between `public.embeddings` + and the wiki content table depends on exact column names from the + consolidation plan. Resolve at implementation time. +- [ ] How to surface "no results" vs "search failed" — empty array + versus error response. Likely empty array for "no results" and error + for "model mismatch / query embed failed." +- [ ] Pagination — `limit` is in scope; offset / cursor pagination is + not. Add later if it's needed. +- [ ] Caching — query embedding cost is small but non-zero. No caching + in v1. If usage patterns show repeated queries, add later. + +## Phases + +### Phase 0 — Verify dependencies + +- [ ] Consolidation plan's Phase 5 (vault indexing) is complete. + `public.embeddings` has rows where `source_schema = 'wiki'` covering + the full vault corpus. +- [ ] HNSW index exists on `public.embeddings.embedding`. +- [ ] Embedding model is documented and reachable from the vault MCP + server's host. +- [ ] The vault MCP user has read access to `public.embeddings`. + +### Phase 1 — Tool implementation + +- [ ] Add `search` function to the vault MCP server. +- [ ] Wire to the configured embedding model for query embedding. +- [ ] Build the parameterized SQL query (vector search + filters). +- [ ] Define the response shape per the locked decisions. +- [ ] Add input validation (filter values exist in schema, date format + correct, etc.). + +### Phase 2 — Verification + +- [ ] Smoke test: search for a topic that's definitely in the vault + ("MCP", "Traefik") — confirm relevant results appear in top 5. +- [ ] Negative test: search for nonsense gibberish — confirm low scores + or empty results, not garbage matches. +- [ ] Filter test: search with `tags=["mcp"]` — confirm results are + restricted correctly. +- [ ] Combined filter test: query + tag + date range — confirm AND + semantics work. +- [ ] Model-mismatch test: temporarily configure a different embedding + model, confirm tool errors rather than returning bad rankings. + +### Phase 3 — Documentation + +- [ ] Add usage examples to the vault MCP server's README. +- [ ] Note the tool in CLAUDE.md so agents know to use it. +- [ ] Capture lessons learned in a session note. + +## Notes + +### Why this isn't part of the consolidation plan + +The consolidation plan builds the substrate — embeddings table, vault +indexing, HNSW index. This plan builds the user-facing tool that +consumes the substrate. Splitting them keeps the consolidation plan +focused on infrastructure and lets the search tool ship with its own +small, focused scope. + +### Why minimal output instead of full content + +Most searches are exploratory — "what do I have on X" — and only a +fraction of returned hits will actually be read. Returning bodies in +the search response means every search pays for content fetching that +mostly isn't used. Keeping search lightweight and giving callers a +separate `get_project` path matches the actual usage pattern. + +### Why a single tool with parameters instead of separate tools + +A single parameterized tool (`vault:search` with optional tags, domain, +etc.) is more flexible than many tools (`vault:search_by_tag`, +`vault:search_recent`, etc.) and matches how natural-language queries +combine criteria. "Find recent context-window notes tagged claude-code" +is one tool call with three filters, not three tool calls composed +manually. + +### Why server-side embedding is the only correct choice + +Each embedding model produces vectors in its own geometry. Vectors from +different models aren't comparable, and a model mismatch produces +silently wrong rankings — not an error, just bad results. The server +owns the stored vectors and knows which model produced them, so it's +the right place to ensure the query uses the same model. Letting +clients embed queries themselves would require every client to know +which model is correct and would fail invisibly when wrong. + +### Future extension: scope parameter + +Anticipated shape for the unified memory search that follows: + +``` +vault:search(query, scope="vault" | "memory" | "all", ...) +``` + +- `scope="vault"` — current behavior, wiki schema only (default) +- `scope="memory"` — ob1.thoughts and ob1.reflections only +- `scope="all"` — everything + +Implementation extends the WHERE clause to include additional +`source_schema` values. The output shape grows a `source` field so +callers can tell vault hits from memory hits. Deferred to a separate +plan once real vault-search usage informs the design. + +### Flag for Lovebug + +The Postgres consolidation plan (`postgres-consolidation-reflection-layer`, +saved 2026-05-15) was written via the now-deprecated `save_note` path. +The vault markdown file exists at +`Sources/Homelab/postgres-consolidation-ob1-reflection-layer.md`, but +the Postgres row does not — `vault:get_project` returns "not found" for +that slug. The plan needs to be re-registered via `vault:create_artifact` +so it shows up in `list_projects`, can have its status updated, and +participates in the Trello card projection. Not blocking for this +search plan, but worth handling before the consolidation plan goes +into execution tracking. \ No newline at end of file