wiki-vault/Sources/Homelab/vault-semantic-search.md

created	path	project	tags	type
2026-05-17	Sources/Homelab	vault-semantic-search	mcp pgvector embeddings wiki	project-plan

Vault Semantic Search

Goal

Add a vault:search MCP tool that takes a query string (plus optional metadata filters) and returns vault notes ranked by semantic similarity. The tool runs entirely against the wiki schema and reuses the embedding infrastructure built by the Postgres consolidation plan.

Concrete use case: "search the vault for context" returns relevant notes without needing to remember slugs or tag names. Semantic search replaces the current "list everything, filter client-side" pattern.

This plan is scoped to vault search only. A future unified memory search spanning vault + thoughts + reflections is anticipated via a scope parameter, but is deferred until vault search is in use and shaping requirements are clearer.

Locked Decisions

Tool surface

• Name: vault:search
• Read-only. No writes, no side effects.
• Single MCP tool — no separate "by tag" or "by date" variants. All filtering is parameterized on this one tool.

Inputs

• query (required, string) — natural-language search query
• tags (optional, list of strings) — restrict to notes carrying any of these tags
• domain (optional, string) — one of the schema's domain values (homelab, dev, venture, reference)
• status (optional, list of strings) — one or more lifecycle statuses
• note_type (optional, string) — restrict to project-plan, session-notes, entity, topic-landing, or synthesis
• date_range (optional, object with start and end) — filter by updated_at timestamp range
• limit (optional, int, default 10) — max results to return

Filters compose via AND. Vector similarity ranks within the filtered candidate set.

Output shape (minimal)

[
  {
    "slug": "ob1-deployment",
    "title": "OB1 Main Deployment",
    "path": "Sources/Homelab/ob1-deployment.md",
    "score": 0.82
  },
  ...
]

Score is the cosine similarity (or pgvector distance converted to similarity — implementation detail). Caller uses vault:get_project to fetch full content for any result they want to read.

Keeping content fetch separate from search keeps the search response cheap and lets callers decide when full body content is worth the round trip.

Embedding behavior

• Query embedding runs server-side inside the vault MCP server.
• The server uses the same embedding model that produced the stored vectors in public.embeddings (recorded in the embedding_model column).
• Mismatches are not silently tolerated — if the configured query model doesn't match the model recorded on the rows being searched, the search returns an error rather than incorrect rankings.
• Embedding model selection is inherited from the consolidation plan; this plan doesn't pick or change models.

Scope

• Searches the wiki schema only. Both Sources and Wiki layers are searchable; filter via note_type if a caller wants Sources-only or Wiki-only.
• Does not search ob1.thoughts or ob1.reflections. That's a separate tool (or this tool extended with a scope parameter) in a future plan.

Backend mechanics

• Vector similarity query against public.embeddings filtered by source_schema = 'wiki'.
• Filters applied via WHERE clauses joining public.embeddings to wiki.notes (or whatever the canonical wiki content table is named).
• HNSW index on public.embeddings.embedding is the consolidation plan's responsibility; this plan reuses it.

Open Items

• Final SQL query shape — JOIN pattern between public.embeddings and the wiki content table depends on exact column names from the consolidation plan. Resolve at implementation time.
• How to surface "no results" vs "search failed" — empty array versus error response. Likely empty array for "no results" and error for "model mismatch / query embed failed."
• Pagination — limit is in scope; offset / cursor pagination is not. Add later if it's needed.
• Caching — query embedding cost is small but non-zero. No caching in v1. If usage patterns show repeated queries, add later.

Phases

Phase 0 — Verify dependencies

• Consolidation plan's Phase 5 (vault indexing) is complete. public.embeddings has rows where source_schema = 'wiki' covering the full vault corpus.
• HNSW index exists on public.embeddings.embedding.
• Embedding model is documented and reachable from the vault MCP server's host.
• The vault MCP user has read access to public.embeddings.

Phase 1 — Tool implementation

• Add search function to the vault MCP server.
• Wire to the configured embedding model for query embedding.
• Build the parameterized SQL query (vector search + filters).
• Define the response shape per the locked decisions.
• Add input validation (filter values exist in schema, date format correct, etc.).

Phase 2 — Verification

• Smoke test: search for a topic that's definitely in the vault ("MCP", "Traefik") — confirm relevant results appear in top 5.
• Negative test: search for nonsense gibberish — confirm low scores or empty results, not garbage matches.
• Filter test: search with tags=["mcp"] — confirm results are restricted correctly.
• Combined filter test: query + tag + date range — confirm AND semantics work.
• Model-mismatch test: temporarily configure a different embedding model, confirm tool errors rather than returning bad rankings.

Phase 3 — Documentation

• Add usage examples to the vault MCP server's README.
• Note the tool in CLAUDE.md so agents know to use it.
• Capture lessons learned in a session note.

Notes

Why this isn't part of the consolidation plan

The consolidation plan builds the substrate — embeddings table, vault indexing, HNSW index. This plan builds the user-facing tool that consumes the substrate. Splitting them keeps the consolidation plan focused on infrastructure and lets the search tool ship with its own small, focused scope.

Why minimal output instead of full content

Most searches are exploratory — "what do I have on X" — and only a fraction of returned hits will actually be read. Returning bodies in the search response means every search pays for content fetching that mostly isn't used. Keeping search lightweight and giving callers a separate get_project path matches the actual usage pattern.

Why a single tool with parameters instead of separate tools

A single parameterized tool (vault:search with optional tags, domain, etc.) is more flexible than many tools (vault:search_by_tag, vault:search_recent, etc.) and matches how natural-language queries combine criteria. "Find recent context-window notes tagged claude-code" is one tool call with three filters, not three tool calls composed manually.

Why server-side embedding is the only correct choice

Each embedding model produces vectors in its own geometry. Vectors from different models aren't comparable, and a model mismatch produces silently wrong rankings — not an error, just bad results. The server owns the stored vectors and knows which model produced them, so it's the right place to ensure the query uses the same model. Letting clients embed queries themselves would require every client to know which model is correct and would fail invisibly when wrong.

Future extension: scope parameter

Anticipated shape for the unified memory search that follows:

vault:search(query, scope="vault" | "memory" | "all", ...)

• scope="vault" — current behavior, wiki schema only (default)
• scope="memory" — ob1.thoughts and ob1.reflections only
• scope="all" — everything

Implementation extends the WHERE clause to include additional source_schema values. The output shape grows a source field so callers can tell vault hits from memory hits. Deferred to a separate plan once real vault-search usage informs the design.

Flag for Lovebug

The Postgres consolidation plan (postgres-consolidation-reflection-layer, saved 2026-05-15) was written via the now-deprecated save_note path. The vault markdown file exists at Sources/Homelab/postgres-consolidation-ob1-reflection-layer.md, but the Postgres row does not — vault:get_project returns "not found" for that slug. The plan needs to be re-registered via vault:create_artifact so it shows up in list_projects, can have its status updated, and participates in the Trello card projection. Not blocking for this search plan, but worth handling before the consolidation plan goes into execution tracking.