mcp: project-plan — Vault Semantic Search
This commit is contained in:
Lovebug MCP
parent
a2d5331738
commit
85130079cb
226
Sources/Homelab/vault-semantic-search.md
Normal file
226
Sources/Homelab/vault-semantic-search.md
Normal file
@ -0,0 +1,226 @@
|
||||
---
|
||||
created: '2026-05-17'
|
||||
path: Sources/Homelab
|
||||
project: vault-semantic-search
|
||||
tags:
|
||||
- mcp
|
||||
- pgvector
|
||||
- embeddings
|
||||
- wiki
|
||||
type: project-plan
|
||||
---
|
||||
|
||||
# Vault Semantic Search
|
||||
|
||||
## Goal
|
||||
|
||||
Add a `vault:search` MCP tool that takes a query string (plus optional
|
||||
metadata filters) and returns vault notes ranked by semantic similarity.
|
||||
The tool runs entirely against the `wiki` schema and reuses the embedding
|
||||
infrastructure built by the Postgres consolidation plan.
|
||||
|
||||
Concrete use case: "search the vault for context" returns relevant notes
|
||||
without needing to remember slugs or tag names. Semantic search replaces
|
||||
the current "list everything, filter client-side" pattern.
|
||||
|
||||
This plan is scoped to vault search only. A future unified memory search
|
||||
spanning vault + thoughts + reflections is anticipated via a `scope`
|
||||
parameter, but is deferred until vault search is in use and shaping
|
||||
requirements are clearer.
|
||||
|
||||
## Locked Decisions
|
||||
|
||||
### Tool surface
|
||||
|
||||
- Name: `vault:search`
|
||||
- Read-only. No writes, no side effects.
|
||||
- Single MCP tool — no separate "by tag" or "by date" variants. All
|
||||
filtering is parameterized on this one tool.
|
||||
|
||||
### Inputs
|
||||
|
||||
- `query` (required, string) — natural-language search query
|
||||
- `tags` (optional, list of strings) — restrict to notes carrying any of
|
||||
these tags
|
||||
- `domain` (optional, string) — one of the schema's domain values
|
||||
(homelab, dev, venture, reference)
|
||||
- `status` (optional, list of strings) — one or more lifecycle statuses
|
||||
- `note_type` (optional, string) — restrict to project-plan,
|
||||
session-notes, entity, topic-landing, or synthesis
|
||||
- `date_range` (optional, object with `start` and `end`) — filter by
|
||||
`updated_at` timestamp range
|
||||
- `limit` (optional, int, default 10) — max results to return
|
||||
|
||||
Filters compose via AND. Vector similarity ranks within the filtered
|
||||
candidate set.
|
||||
|
||||
### Output shape (minimal)
|
||||
|
||||
```
|
||||
[
|
||||
{
|
||||
"slug": "ob1-deployment",
|
||||
"title": "OB1 Main Deployment",
|
||||
"path": "Sources/Homelab/ob1-deployment.md",
|
||||
"score": 0.82
|
||||
},
|
||||
...
|
||||
]
|
||||
```
|
||||
|
||||
Score is the cosine similarity (or pgvector distance converted to
|
||||
similarity — implementation detail). Caller uses `vault:get_project`
|
||||
to fetch full content for any result they want to read.
|
||||
|
||||
Keeping content fetch separate from search keeps the search response
|
||||
cheap and lets callers decide when full body content is worth the round
|
||||
trip.
|
||||
|
||||
### Embedding behavior
|
||||
|
||||
- Query embedding runs **server-side** inside the vault MCP server.
|
||||
- The server uses the **same embedding model** that produced the stored
|
||||
vectors in `public.embeddings` (recorded in the `embedding_model`
|
||||
column).
|
||||
- Mismatches are not silently tolerated — if the configured query model
|
||||
doesn't match the model recorded on the rows being searched, the
|
||||
search returns an error rather than incorrect rankings.
|
||||
- Embedding model selection is inherited from the consolidation plan;
|
||||
this plan doesn't pick or change models.
|
||||
|
||||
### Scope
|
||||
|
||||
- Searches the `wiki` schema only. Both Sources and Wiki layers are
|
||||
searchable; filter via `note_type` if a caller wants Sources-only or
|
||||
Wiki-only.
|
||||
- Does not search `ob1.thoughts` or `ob1.reflections`. That's a separate
|
||||
tool (or this tool extended with a `scope` parameter) in a future plan.
|
||||
|
||||
### Backend mechanics
|
||||
|
||||
- Vector similarity query against `public.embeddings` filtered by
|
||||
`source_schema = 'wiki'`.
|
||||
- Filters applied via WHERE clauses joining `public.embeddings` to
|
||||
`wiki.notes` (or whatever the canonical wiki content table is named).
|
||||
- HNSW index on `public.embeddings.embedding` is the consolidation
|
||||
plan's responsibility; this plan reuses it.
|
||||
|
||||
## Open Items
|
||||
|
||||
- [ ] Final SQL query shape — JOIN pattern between `public.embeddings`
|
||||
and the wiki content table depends on exact column names from the
|
||||
consolidation plan. Resolve at implementation time.
|
||||
- [ ] How to surface "no results" vs "search failed" — empty array
|
||||
versus error response. Likely empty array for "no results" and error
|
||||
for "model mismatch / query embed failed."
|
||||
- [ ] Pagination — `limit` is in scope; offset / cursor pagination is
|
||||
not. Add later if it's needed.
|
||||
- [ ] Caching — query embedding cost is small but non-zero. No caching
|
||||
in v1. If usage patterns show repeated queries, add later.
|
||||
|
||||
## Phases
|
||||
|
||||
### Phase 0 — Verify dependencies
|
||||
|
||||
- [ ] Consolidation plan's Phase 5 (vault indexing) is complete.
|
||||
`public.embeddings` has rows where `source_schema = 'wiki'` covering
|
||||
the full vault corpus.
|
||||
- [ ] HNSW index exists on `public.embeddings.embedding`.
|
||||
- [ ] Embedding model is documented and reachable from the vault MCP
|
||||
server's host.
|
||||
- [ ] The vault MCP user has read access to `public.embeddings`.
|
||||
|
||||
### Phase 1 — Tool implementation
|
||||
|
||||
- [ ] Add `search` function to the vault MCP server.
|
||||
- [ ] Wire to the configured embedding model for query embedding.
|
||||
- [ ] Build the parameterized SQL query (vector search + filters).
|
||||
- [ ] Define the response shape per the locked decisions.
|
||||
- [ ] Add input validation (filter values exist in schema, date format
|
||||
correct, etc.).
|
||||
|
||||
### Phase 2 — Verification
|
||||
|
||||
- [ ] Smoke test: search for a topic that's definitely in the vault
|
||||
("MCP", "Traefik") — confirm relevant results appear in top 5.
|
||||
- [ ] Negative test: search for nonsense gibberish — confirm low scores
|
||||
or empty results, not garbage matches.
|
||||
- [ ] Filter test: search with `tags=["mcp"]` — confirm results are
|
||||
restricted correctly.
|
||||
- [ ] Combined filter test: query + tag + date range — confirm AND
|
||||
semantics work.
|
||||
- [ ] Model-mismatch test: temporarily configure a different embedding
|
||||
model, confirm tool errors rather than returning bad rankings.
|
||||
|
||||
### Phase 3 — Documentation
|
||||
|
||||
- [ ] Add usage examples to the vault MCP server's README.
|
||||
- [ ] Note the tool in CLAUDE.md so agents know to use it.
|
||||
- [ ] Capture lessons learned in a session note.
|
||||
|
||||
## Notes
|
||||
|
||||
### Why this isn't part of the consolidation plan
|
||||
|
||||
The consolidation plan builds the substrate — embeddings table, vault
|
||||
indexing, HNSW index. This plan builds the user-facing tool that
|
||||
consumes the substrate. Splitting them keeps the consolidation plan
|
||||
focused on infrastructure and lets the search tool ship with its own
|
||||
small, focused scope.
|
||||
|
||||
### Why minimal output instead of full content
|
||||
|
||||
Most searches are exploratory — "what do I have on X" — and only a
|
||||
fraction of returned hits will actually be read. Returning bodies in
|
||||
the search response means every search pays for content fetching that
|
||||
mostly isn't used. Keeping search lightweight and giving callers a
|
||||
separate `get_project` path matches the actual usage pattern.
|
||||
|
||||
### Why a single tool with parameters instead of separate tools
|
||||
|
||||
A single parameterized tool (`vault:search` with optional tags, domain,
|
||||
etc.) is more flexible than many tools (`vault:search_by_tag`,
|
||||
`vault:search_recent`, etc.) and matches how natural-language queries
|
||||
combine criteria. "Find recent context-window notes tagged claude-code"
|
||||
is one tool call with three filters, not three tool calls composed
|
||||
manually.
|
||||
|
||||
### Why server-side embedding is the only correct choice
|
||||
|
||||
Each embedding model produces vectors in its own geometry. Vectors from
|
||||
different models aren't comparable, and a model mismatch produces
|
||||
silently wrong rankings — not an error, just bad results. The server
|
||||
owns the stored vectors and knows which model produced them, so it's
|
||||
the right place to ensure the query uses the same model. Letting
|
||||
clients embed queries themselves would require every client to know
|
||||
which model is correct and would fail invisibly when wrong.
|
||||
|
||||
### Future extension: scope parameter
|
||||
|
||||
Anticipated shape for the unified memory search that follows:
|
||||
|
||||
```
|
||||
vault:search(query, scope="vault" | "memory" | "all", ...)
|
||||
```
|
||||
|
||||
- `scope="vault"` — current behavior, wiki schema only (default)
|
||||
- `scope="memory"` — ob1.thoughts and ob1.reflections only
|
||||
- `scope="all"` — everything
|
||||
|
||||
Implementation extends the WHERE clause to include additional
|
||||
`source_schema` values. The output shape grows a `source` field so
|
||||
callers can tell vault hits from memory hits. Deferred to a separate
|
||||
plan once real vault-search usage informs the design.
|
||||
|
||||
### Flag for Lovebug
|
||||
|
||||
The Postgres consolidation plan (`postgres-consolidation-reflection-layer`,
|
||||
saved 2026-05-15) was written via the now-deprecated `save_note` path.
|
||||
The vault markdown file exists at
|
||||
`Sources/Homelab/postgres-consolidation-ob1-reflection-layer.md`, but
|
||||
the Postgres row does not — `vault:get_project` returns "not found" for
|
||||
that slug. The plan needs to be re-registered via `vault:create_artifact`
|
||||
so it shows up in `list_projects`, can have its status updated, and
|
||||
participates in the Trello card projection. Not blocking for this
|
||||
search plan, but worth handling before the consolidation plan goes
|
||||
into execution tracking.
|
||||
Loading…
Reference in New Issue
Block a user