mcp: project-plan — PBS Hub — Vector Search
This commit is contained in:
Lovebug MCP
parent
0e661bbf86
commit
c634612ad2
209
Sources/Dev/pbs-hub-vector-search.md
Normal file
209
Sources/Dev/pbs-hub-vector-search.md
Normal file
@ -0,0 +1,209 @@
|
|||||||
|
---
|
||||||
|
created: '2026-05-19'
|
||||||
|
path: Sources/Dev
|
||||||
|
project: pbs-hub-vector-search
|
||||||
|
tags:
|
||||||
|
- pbs
|
||||||
|
- pbs-hub
|
||||||
|
- mcp
|
||||||
|
- vector-search
|
||||||
|
- embeddings
|
||||||
|
- lancedb
|
||||||
|
- n8n
|
||||||
|
- openai
|
||||||
|
- rag
|
||||||
|
type: project-plan
|
||||||
|
---
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
|
||||||
|
Add semantic vector search to `pbs-hub-mcp`'s `search_projects` tool, replacing the v1 title-only implementation. Lets an LLM (or human via paste-import-style search UI later) find projects by *meaning* rather than by exact title match — "find me the cobbler video where Jenny talks about gluten substitution" instead of needing to remember the title.
|
||||||
|
|
||||||
|
This is the Phase 4 deferred work from `pbs-hub-data-import-and-mcp`. Pulled out into its own plan because the design conversation got substantial enough to warrant its own home, *and* because the feature isn't needed until PBS has accumulated enough content for title search to feel limiting. At one video/week that's probably 18+ months out.
|
||||||
|
|
||||||
|
**Trigger to start build:** title search on the real PBS project library has become a real friction point. Until then this plan sits as documented direction.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Locked Decisions (Preliminary)
|
||||||
|
|
||||||
|
These represent the leaning state from design discussion. To be confirmed at Phase 4 kickoff with whatever's true at that point in time (model availability, infra state, etc.).
|
||||||
|
|
||||||
|
### Vector store
|
||||||
|
|
||||||
|
**LanceDB** is the leaning choice. Alternative considered: `sqlite-vec`. Both are embedded (no separate container needed), both fit comfortably at PBS scale. LanceDB's columnar storage, versioning, and modern AI/RAG ecosystem positioning give it the edge. Either is defensible.
|
||||||
|
|
||||||
|
Open at kickoff: confirm LanceDB hasn't been superseded by something better; confirm Python ecosystem maturity is still strong.
|
||||||
|
|
||||||
|
### Embedding model
|
||||||
|
|
||||||
|
**OpenAI `text-embedding-3-small`** (or whatever its successor is at Phase 4 kickoff).
|
||||||
|
|
||||||
|
Reasons:
|
||||||
|
- Cost is negligible at PBS scale ($0.02/M tokens — entire library embedding is likely cents total)
|
||||||
|
- Mature, stable, multi-year API lifecycle
|
||||||
|
- Wide tooling support; every vector DB integrates natively
|
||||||
|
- Default 1536 dimensions is plenty for retrieval quality at this scale
|
||||||
|
|
||||||
|
**Embedding model must remain consistent.** All vectors in the index must be generated by the same model. Switching models = re-embedding everything. Cost of re-embedding is trivial; the *operational discipline* of "never silently mix models in the same index" is what matters.
|
||||||
|
|
||||||
|
**Store model name + version alongside each embedding** as a column in LanceDB. Makes mixed-model bugs impossible to introduce silently and makes migration trivially debuggable.
|
||||||
|
|
||||||
|
### Embedding generation location
|
||||||
|
|
||||||
|
**Async worker pattern, n8n-driven.**
|
||||||
|
|
||||||
|
Architecture:
|
||||||
|
```
|
||||||
|
pbs-hub (pbs Linode) — content changes
|
||||||
|
↓ webhook
|
||||||
|
n8n (herbydev) — picks up event
|
||||||
|
↓ calls embedding API
|
||||||
|
OpenAI (or whichever provider)
|
||||||
|
↓ writes vector
|
||||||
|
LanceDB (likely on pbs Linode)
|
||||||
|
↑ queried by
|
||||||
|
pbs-hub-mcp (pbs Linode) — when LLM client searches
|
||||||
|
```
|
||||||
|
|
||||||
|
Why this shape:
|
||||||
|
- Embedding latency doesn't block user-facing writes in pbs-hub
|
||||||
|
- pbs-hub stays focused on CMS work; no embedding logic in the request path
|
||||||
|
- n8n is already the right hammer for "when X happens, do Y" — pattern is reusable
|
||||||
|
- Failure isolation: embedding gen failure doesn't lose user content
|
||||||
|
- Eventual consistency is fine (5-second lag between save and searchability is invisible)
|
||||||
|
|
||||||
|
### Vector store physical location
|
||||||
|
|
||||||
|
**LanceDB on the pbs Linode.** Reasoning:
|
||||||
|
- pbs-hub-mcp queries it on every search → read latency matters
|
||||||
|
- Writes are async/infrequent (whenever content changes) → write latency tolerable
|
||||||
|
- Backups go alongside other pbs Linode data
|
||||||
|
- Cross-site reads (MCP on Linode reading from herbydev) would be slower than cross-site writes (n8n on herbydev writing to Linode)
|
||||||
|
|
||||||
|
n8n on herbydev writes across to the pbs Linode's LanceDB volume. pbs-hub-mcp reads locally on Linode.
|
||||||
|
|
||||||
|
### Embedding units (what gets embedded)
|
||||||
|
|
||||||
|
**Per-scene rollup as the primary unit**, with per-project-field rollup for top-level fields.
|
||||||
|
|
||||||
|
- **Project-level**: one embedding for `outline`, one for `treatment`. Each metadata value embedded separately if it's substantive (logline yes, tagline probably not — decide at kickoff).
|
||||||
|
- **Scene-level**: one embedding per scene, content = concatenated `purpose` + `in_shot` + `background` + `script` + scene item labels.
|
||||||
|
- **Chunking**: only when a content unit exceeds ~3500 tokens (well under the 8191 ceiling of `text-embedding-3-small`). Overlap ~300 tokens between chunks when chunking is needed.
|
||||||
|
|
||||||
|
Reasoning:
|
||||||
|
- Matches mental model of search: "find the cobbler scene about substitution" → return a scene, not a paragraph
|
||||||
|
- At PBS scale, almost no scene rollup will exceed 3500 tokens. Chunking is the rare case, not the default
|
||||||
|
- Storing source reference + chunk index alongside each embedding means search results can name what they came from ("scene 3 of Blueberry Cobbler, chunk 0")
|
||||||
|
- Easy to add per-field precision later (Phase 5+) without throwing away the per-scene index
|
||||||
|
|
||||||
|
### `search_projects` tool surface
|
||||||
|
|
||||||
|
Replaces the v1 title-only behavior. The MCP tool surface stays the same shape — query in, results out — but the *body* of the search uses vectors.
|
||||||
|
|
||||||
|
- Accepts same `query` parameter (free text)
|
||||||
|
- Returns same result shape (project summaries with relevance score)
|
||||||
|
- Internally: embed the query with the same model, compare against indexed vectors, return top N
|
||||||
|
- Optional: hybrid ranking (combine vector similarity with title BM25). Stretch goal at Phase 4 kickoff; pure vector is fine for v1.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Open Items (revisit at Phase 4 kickoff)
|
||||||
|
|
||||||
|
- [ ] **Vector store final pick** — LanceDB still the leading option at kickoff time, or has the ecosystem moved? Sanity-check `sqlite-vec` and any new entrants.
|
||||||
|
- [ ] **Embedding model final pick** — confirm `text-embedding-3-small` is still current, not deprecated; consider Voyage or Cohere if pricing or quality has shifted.
|
||||||
|
- [ ] **Embedding generation: hosted vs local** — at Phase 4 kickoff, reconsider hosted-vs-local. Local (Ollama + nomic-embed-text on herbydev) was discussed but hosted won on grounds of cost being negligible and operational simplicity. Worth a fresh look if homelab AI infra has matured.
|
||||||
|
- [ ] **Tagline embedding** — embed tagline as its own vector, or skip (too short to be useful)? Decide once real metadata patterns emerge.
|
||||||
|
- [ ] **Hybrid ranking** — pure vector vs BM25+vector hybrid. Pure vector is v1; hybrid is stretch goal or future enhancement.
|
||||||
|
- [ ] **Dimensions config** — `text-embedding-3-small` default is 1536; configurable down to 512 for storage savings. At PBS scale, default is fine. Reconsider if storage becomes a concern.
|
||||||
|
- [ ] **LanceDB volume strategy** — docker volume mount? Where does it live on the host? Backup story? Surface for Lovebug at kickoff.
|
||||||
|
- [ ] **Re-embedding workflow** — script or n8n flow to regenerate all embeddings when model changes or chunking strategy changes. Worth designing once Phase 4 starts.
|
||||||
|
- [ ] **Search ranking transparency** — return raw similarity scores to the LLM, or thresholded boolean ("yes this matched")? Affects how the LLM reasons about result confidence.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phases
|
||||||
|
|
||||||
|
### Phase 1 — Vector store + embedding pipeline foundation
|
||||||
|
|
||||||
|
- [ ] LanceDB deployed on pbs Linode (or alternative if reconsidered at kickoff)
|
||||||
|
- [ ] Docker volume strategy decided and implemented
|
||||||
|
- [ ] Backup approach defined
|
||||||
|
- [ ] n8n flow for "content change → embed → write to LanceDB"
|
||||||
|
- [ ] Webhook receiver from pbs-hub
|
||||||
|
- [ ] Embedding API call (OpenAI or alternative)
|
||||||
|
- [ ] Write to LanceDB
|
||||||
|
- [ ] Error handling + retry logic
|
||||||
|
- [ ] pbs-hub webhook firing on content changes
|
||||||
|
- [ ] Fire on Project.outline / treatment / metadata save
|
||||||
|
- [ ] Fire on Scene save (any briefing or script field)
|
||||||
|
- [ ] Fire on SceneItem save (label change)
|
||||||
|
- [ ] Debouncing if rapid edits cause webhook storm
|
||||||
|
- [ ] LanceDB schema: vector + source reference + chunk index + model name + model version + timestamp
|
||||||
|
|
||||||
|
### Phase 2 — Initial backfill
|
||||||
|
|
||||||
|
- [ ] Script to embed all existing PBS content from current state of MySQL
|
||||||
|
- [ ] Run once to populate LanceDB
|
||||||
|
- [ ] Document the script for future re-embedding when model changes
|
||||||
|
|
||||||
|
### Phase 3 — `search_projects` MCP tool replacement
|
||||||
|
|
||||||
|
- [ ] Update pbs-hub-mcp's `search_projects` tool to use LanceDB
|
||||||
|
- [ ] Internally: embed query with same model, vector similarity search, return top N
|
||||||
|
- [ ] Return shape matches v1 (project summaries with score)
|
||||||
|
- [ ] Integration test: known content returns expected matches
|
||||||
|
- [ ] Deploy to staging, validate against real searches
|
||||||
|
- [ ] Deploy to prod
|
||||||
|
|
||||||
|
### Phase 4 — Polish and learning loop
|
||||||
|
|
||||||
|
- [ ] Monitor real search behavior — which queries return what
|
||||||
|
- [ ] Adjust chunking thresholds based on observed content sizes
|
||||||
|
- [ ] Consider hybrid ranking if pure vector misses obvious title hits
|
||||||
|
- [ ] Re-evaluate embedding model based on real-world result quality
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
|
||||||
|
### Why this is its own plan
|
||||||
|
|
||||||
|
The vector search design conversation expanded beyond what would fit in `pbs-hub-data-import-and-mcp`'s Phase 4 placeholder. Key design choices (embedding model, generation location, chunking strategy) all interact and benefit from explicit documentation. Spinning it out preserves that thinking for future-you.
|
||||||
|
|
||||||
|
### Scale check
|
||||||
|
|
||||||
|
At PBS's actual scale (one video/week, ~50/year), title-only search remains effective for the foreseeable future. This plan stays parked until title search becomes a real pain point. Realistic trigger window: 18+ months of accumulated content, or whenever Travis or Jenny says "I can't find that video about the thing where she talked about substitutions."
|
||||||
|
|
||||||
|
### Cross-references
|
||||||
|
|
||||||
|
- **Parent plan:** `pbs-hub-data-import-and-mcp` — this is its Phase 4 extracted into a standalone plan
|
||||||
|
- **Dependencies:**
|
||||||
|
- `pbs-hub-scene-management` schema must be live (content to embed)
|
||||||
|
- `pbs-hub-data-import-and-mcp` Phase 2 must be live (MCP server to update)
|
||||||
|
- n8n on herbydev (already running) for the embedding worker
|
||||||
|
- LanceDB / vector store infra (new for this work)
|
||||||
|
|
||||||
|
### Key design rules to preserve
|
||||||
|
|
||||||
|
- **Never mix embedding models in one index.** Store model name + version with every vector. Migration = full re-embed.
|
||||||
|
- **Embedding generation is async, not request-blocking.** pbs-hub never waits on an embedding API call.
|
||||||
|
- **Chunking is the rare case at PBS scale, not the default.** Per-scene rollup as the standard unit; chunk only when content exceeds ~3500 tokens.
|
||||||
|
- **Eventual consistency is fine.** A few seconds of lag between content save and searchability is invisible.
|
||||||
|
|
||||||
|
### Flagged for plan readers
|
||||||
|
|
||||||
|
- This plan is **not on the immediate critical path.** Don't pull it forward without a real pain point. Premature build wastes time tuning a system against a too-small corpus.
|
||||||
|
- The embedding model is the *workhorse* of the system. If you ever need to change it, plan an afternoon for re-embedding the whole library. Trivial at PBS scale, but real work.
|
||||||
|
- LanceDB / sqlite-vec / wherever-the-ecosystem-lands at Phase 4 kickoff — don't lock in the choice from this plan without sanity-checking what's current at that point.
|
||||||
|
|
||||||
|
### Conversation context
|
||||||
|
|
||||||
|
This plan emerged from a chain of "while we're here..." extensions to the original scene-management work:
|
||||||
|
- Started as a Phase 4 stub in `pbs-hub-data-import-and-mcp`
|
||||||
|
- Travis flagged interest in embedded vector DBs (originally LanceDB)
|
||||||
|
- Conversation surfaced real architecture questions (embedding generation location, model choice, chunking strategy)
|
||||||
|
- Cost analysis showed hosted embeddings are cents at PBS scale — Travis chose hosted over local
|
||||||
|
- Travis recognized scale realities ("it's probably going to take me a while to generate enough video ideas") and asked to defer with documented direction
|
||||||
|
|
||||||
|
That last beat is the right one. Plan filed, build deferred.
|
||||||
Loading…
Reference in New Issue
Block a user