--- created: '2026-05-17' path: Sources/Homelab project: vault-semantic-search tags: - mcp - pgvector - embeddings - wiki type: project-plan --- # Vault Semantic Search ## Goal Add a `vault:search` MCP tool that takes a query string (plus optional metadata filters) and returns vault notes ranked by semantic similarity. The tool runs entirely against the `wiki` schema and reuses the embedding infrastructure built by the Postgres consolidation plan. Concrete use case: "search the vault for context" returns relevant notes without needing to remember slugs or tag names. Semantic search replaces the current "list everything, filter client-side" pattern. This plan is scoped to vault search only. A future unified memory search spanning vault + thoughts + reflections is anticipated via a `scope` parameter, but is deferred until vault search is in use and shaping requirements are clearer. ## Locked Decisions ### Tool surface - Name: `vault:search` - Read-only. No writes, no side effects. - Single MCP tool — no separate "by tag" or "by date" variants. All filtering is parameterized on this one tool. ### Inputs - `query` (required, string) — natural-language search query - `tags` (optional, list of strings) — restrict to notes carrying any of these tags - `domain` (optional, string) — one of the schema's domain values (homelab, dev, venture, reference) - `status` (optional, list of strings) — one or more lifecycle statuses - `note_type` (optional, string) — restrict to project-plan, session-notes, entity, topic-landing, or synthesis - `date_range` (optional, object with `start` and `end`) — filter by `updated_at` timestamp range - `limit` (optional, int, default 10) — max results to return Filters compose via AND. Vector similarity ranks within the filtered candidate set. ### Output shape (minimal) ``` [ { "slug": "ob1-deployment", "title": "OB1 Main Deployment", "path": "Sources/Homelab/ob1-deployment.md", "score": 0.82 }, ... ] ``` Score is the cosine similarity (or pgvector distance converted to similarity — implementation detail). Caller uses `vault:get_project` to fetch full content for any result they want to read. Keeping content fetch separate from search keeps the search response cheap and lets callers decide when full body content is worth the round trip. ### Embedding behavior - Query embedding runs **server-side** inside the vault MCP server. - The server uses the **same embedding model** that produced the stored vectors in `public.embeddings` (recorded in the `embedding_model` column). - Mismatches are not silently tolerated — if the configured query model doesn't match the model recorded on the rows being searched, the search returns an error rather than incorrect rankings. - Embedding model selection is inherited from the consolidation plan; this plan doesn't pick or change models. ### Scope - Searches the `wiki` schema only. Both Sources and Wiki layers are searchable; filter via `note_type` if a caller wants Sources-only or Wiki-only. - Does not search `ob1.thoughts` or `ob1.reflections`. That's a separate tool (or this tool extended with a `scope` parameter) in a future plan. ### Backend mechanics - Vector similarity query against `public.embeddings` filtered by `source_schema = 'wiki'`. - Filters applied via WHERE clauses joining `public.embeddings` to `wiki.notes` (or whatever the canonical wiki content table is named). - HNSW index on `public.embeddings.embedding` is the consolidation plan's responsibility; this plan reuses it. ## Open Items - [ ] Final SQL query shape — JOIN pattern between `public.embeddings` and the wiki content table depends on exact column names from the consolidation plan. Resolve at implementation time. - [ ] How to surface "no results" vs "search failed" — empty array versus error response. Likely empty array for "no results" and error for "model mismatch / query embed failed." - [ ] Pagination — `limit` is in scope; offset / cursor pagination is not. Add later if it's needed. - [ ] Caching — query embedding cost is small but non-zero. No caching in v1. If usage patterns show repeated queries, add later. ## Phases ### Phase 0 — Verify dependencies - [ ] Consolidation plan's Phase 5 (vault indexing) is complete. `public.embeddings` has rows where `source_schema = 'wiki'` covering the full vault corpus. - [ ] HNSW index exists on `public.embeddings.embedding`. - [ ] Embedding model is documented and reachable from the vault MCP server's host. - [ ] The vault MCP user has read access to `public.embeddings`. ### Phase 1 — Tool implementation - [ ] Add `search` function to the vault MCP server. - [ ] Wire to the configured embedding model for query embedding. - [ ] Build the parameterized SQL query (vector search + filters). - [ ] Define the response shape per the locked decisions. - [ ] Add input validation (filter values exist in schema, date format correct, etc.). ### Phase 2 — Verification - [ ] Smoke test: search for a topic that's definitely in the vault ("MCP", "Traefik") — confirm relevant results appear in top 5. - [ ] Negative test: search for nonsense gibberish — confirm low scores or empty results, not garbage matches. - [ ] Filter test: search with `tags=["mcp"]` — confirm results are restricted correctly. - [ ] Combined filter test: query + tag + date range — confirm AND semantics work. - [ ] Model-mismatch test: temporarily configure a different embedding model, confirm tool errors rather than returning bad rankings. ### Phase 3 — Documentation - [ ] Add usage examples to the vault MCP server's README. - [ ] Note the tool in CLAUDE.md so agents know to use it. - [ ] Capture lessons learned in a session note. ## Notes ### Why this isn't part of the consolidation plan The consolidation plan builds the substrate — embeddings table, vault indexing, HNSW index. This plan builds the user-facing tool that consumes the substrate. Splitting them keeps the consolidation plan focused on infrastructure and lets the search tool ship with its own small, focused scope. ### Why minimal output instead of full content Most searches are exploratory — "what do I have on X" — and only a fraction of returned hits will actually be read. Returning bodies in the search response means every search pays for content fetching that mostly isn't used. Keeping search lightweight and giving callers a separate `get_project` path matches the actual usage pattern. ### Why a single tool with parameters instead of separate tools A single parameterized tool (`vault:search` with optional tags, domain, etc.) is more flexible than many tools (`vault:search_by_tag`, `vault:search_recent`, etc.) and matches how natural-language queries combine criteria. "Find recent context-window notes tagged claude-code" is one tool call with three filters, not three tool calls composed manually. ### Why server-side embedding is the only correct choice Each embedding model produces vectors in its own geometry. Vectors from different models aren't comparable, and a model mismatch produces silently wrong rankings — not an error, just bad results. The server owns the stored vectors and knows which model produced them, so it's the right place to ensure the query uses the same model. Letting clients embed queries themselves would require every client to know which model is correct and would fail invisibly when wrong. ### Future extension: scope parameter Anticipated shape for the unified memory search that follows: ``` vault:search(query, scope="vault" | "memory" | "all", ...) ``` - `scope="vault"` — current behavior, wiki schema only (default) - `scope="memory"` — ob1.thoughts and ob1.reflections only - `scope="all"` — everything Implementation extends the WHERE clause to include additional `source_schema` values. The output shape grows a `source` field so callers can tell vault hits from memory hits. Deferred to a separate plan once real vault-search usage informs the design. ### Flag for Lovebug The Postgres consolidation plan (`postgres-consolidation-reflection-layer`, saved 2026-05-15) was written via the now-deprecated `save_note` path. The vault markdown file exists at `Sources/Homelab/postgres-consolidation-ob1-reflection-layer.md`, but the Postgres row does not — `vault:get_project` returns "not found" for that slug. The plan needs to be re-registered via `vault:create_artifact` so it shows up in `list_projects`, can have its status updated, and participates in the Trello card projection. Not blocking for this search plan, but worth handling before the consolidation plan goes into execution tracking.