diff --git a/Sources/Dev/pbs-hub-vector-search.md b/Sources/Dev/pbs-hub-vector-search.md new file mode 100644 index 0000000..7bef550 --- /dev/null +++ b/Sources/Dev/pbs-hub-vector-search.md @@ -0,0 +1,209 @@ +--- +created: '2026-05-19' +path: Sources/Dev +project: pbs-hub-vector-search +tags: +- pbs +- pbs-hub +- mcp +- vector-search +- embeddings +- lancedb +- n8n +- openai +- rag +type: project-plan +--- + +## Goal + +Add semantic vector search to `pbs-hub-mcp`'s `search_projects` tool, replacing the v1 title-only implementation. Lets an LLM (or human via paste-import-style search UI later) find projects by *meaning* rather than by exact title match — "find me the cobbler video where Jenny talks about gluten substitution" instead of needing to remember the title. + +This is the Phase 4 deferred work from `pbs-hub-data-import-and-mcp`. Pulled out into its own plan because the design conversation got substantial enough to warrant its own home, *and* because the feature isn't needed until PBS has accumulated enough content for title search to feel limiting. At one video/week that's probably 18+ months out. + +**Trigger to start build:** title search on the real PBS project library has become a real friction point. Until then this plan sits as documented direction. + +--- + +## Locked Decisions (Preliminary) + +These represent the leaning state from design discussion. To be confirmed at Phase 4 kickoff with whatever's true at that point in time (model availability, infra state, etc.). + +### Vector store + +**LanceDB** is the leaning choice. Alternative considered: `sqlite-vec`. Both are embedded (no separate container needed), both fit comfortably at PBS scale. LanceDB's columnar storage, versioning, and modern AI/RAG ecosystem positioning give it the edge. Either is defensible. + +Open at kickoff: confirm LanceDB hasn't been superseded by something better; confirm Python ecosystem maturity is still strong. + +### Embedding model + +**OpenAI `text-embedding-3-small`** (or whatever its successor is at Phase 4 kickoff). + +Reasons: +- Cost is negligible at PBS scale ($0.02/M tokens — entire library embedding is likely cents total) +- Mature, stable, multi-year API lifecycle +- Wide tooling support; every vector DB integrates natively +- Default 1536 dimensions is plenty for retrieval quality at this scale + +**Embedding model must remain consistent.** All vectors in the index must be generated by the same model. Switching models = re-embedding everything. Cost of re-embedding is trivial; the *operational discipline* of "never silently mix models in the same index" is what matters. + +**Store model name + version alongside each embedding** as a column in LanceDB. Makes mixed-model bugs impossible to introduce silently and makes migration trivially debuggable. + +### Embedding generation location + +**Async worker pattern, n8n-driven.** + +Architecture: +``` +pbs-hub (pbs Linode) — content changes + ↓ webhook +n8n (herbydev) — picks up event + ↓ calls embedding API +OpenAI (or whichever provider) + ↓ writes vector +LanceDB (likely on pbs Linode) + ↑ queried by +pbs-hub-mcp (pbs Linode) — when LLM client searches +``` + +Why this shape: +- Embedding latency doesn't block user-facing writes in pbs-hub +- pbs-hub stays focused on CMS work; no embedding logic in the request path +- n8n is already the right hammer for "when X happens, do Y" — pattern is reusable +- Failure isolation: embedding gen failure doesn't lose user content +- Eventual consistency is fine (5-second lag between save and searchability is invisible) + +### Vector store physical location + +**LanceDB on the pbs Linode.** Reasoning: +- pbs-hub-mcp queries it on every search → read latency matters +- Writes are async/infrequent (whenever content changes) → write latency tolerable +- Backups go alongside other pbs Linode data +- Cross-site reads (MCP on Linode reading from herbydev) would be slower than cross-site writes (n8n on herbydev writing to Linode) + +n8n on herbydev writes across to the pbs Linode's LanceDB volume. pbs-hub-mcp reads locally on Linode. + +### Embedding units (what gets embedded) + +**Per-scene rollup as the primary unit**, with per-project-field rollup for top-level fields. + +- **Project-level**: one embedding for `outline`, one for `treatment`. Each metadata value embedded separately if it's substantive (logline yes, tagline probably not — decide at kickoff). +- **Scene-level**: one embedding per scene, content = concatenated `purpose` + `in_shot` + `background` + `script` + scene item labels. +- **Chunking**: only when a content unit exceeds ~3500 tokens (well under the 8191 ceiling of `text-embedding-3-small`). Overlap ~300 tokens between chunks when chunking is needed. + +Reasoning: +- Matches mental model of search: "find the cobbler scene about substitution" → return a scene, not a paragraph +- At PBS scale, almost no scene rollup will exceed 3500 tokens. Chunking is the rare case, not the default +- Storing source reference + chunk index alongside each embedding means search results can name what they came from ("scene 3 of Blueberry Cobbler, chunk 0") +- Easy to add per-field precision later (Phase 5+) without throwing away the per-scene index + +### `search_projects` tool surface + +Replaces the v1 title-only behavior. The MCP tool surface stays the same shape — query in, results out — but the *body* of the search uses vectors. + +- Accepts same `query` parameter (free text) +- Returns same result shape (project summaries with relevance score) +- Internally: embed the query with the same model, compare against indexed vectors, return top N +- Optional: hybrid ranking (combine vector similarity with title BM25). Stretch goal at Phase 4 kickoff; pure vector is fine for v1. + +--- + +## Open Items (revisit at Phase 4 kickoff) + +- [ ] **Vector store final pick** — LanceDB still the leading option at kickoff time, or has the ecosystem moved? Sanity-check `sqlite-vec` and any new entrants. +- [ ] **Embedding model final pick** — confirm `text-embedding-3-small` is still current, not deprecated; consider Voyage or Cohere if pricing or quality has shifted. +- [ ] **Embedding generation: hosted vs local** — at Phase 4 kickoff, reconsider hosted-vs-local. Local (Ollama + nomic-embed-text on herbydev) was discussed but hosted won on grounds of cost being negligible and operational simplicity. Worth a fresh look if homelab AI infra has matured. +- [ ] **Tagline embedding** — embed tagline as its own vector, or skip (too short to be useful)? Decide once real metadata patterns emerge. +- [ ] **Hybrid ranking** — pure vector vs BM25+vector hybrid. Pure vector is v1; hybrid is stretch goal or future enhancement. +- [ ] **Dimensions config** — `text-embedding-3-small` default is 1536; configurable down to 512 for storage savings. At PBS scale, default is fine. Reconsider if storage becomes a concern. +- [ ] **LanceDB volume strategy** — docker volume mount? Where does it live on the host? Backup story? Surface for Lovebug at kickoff. +- [ ] **Re-embedding workflow** — script or n8n flow to regenerate all embeddings when model changes or chunking strategy changes. Worth designing once Phase 4 starts. +- [ ] **Search ranking transparency** — return raw similarity scores to the LLM, or thresholded boolean ("yes this matched")? Affects how the LLM reasons about result confidence. + +--- + +## Phases + +### Phase 1 — Vector store + embedding pipeline foundation + +- [ ] LanceDB deployed on pbs Linode (or alternative if reconsidered at kickoff) + - [ ] Docker volume strategy decided and implemented + - [ ] Backup approach defined +- [ ] n8n flow for "content change → embed → write to LanceDB" + - [ ] Webhook receiver from pbs-hub + - [ ] Embedding API call (OpenAI or alternative) + - [ ] Write to LanceDB + - [ ] Error handling + retry logic +- [ ] pbs-hub webhook firing on content changes + - [ ] Fire on Project.outline / treatment / metadata save + - [ ] Fire on Scene save (any briefing or script field) + - [ ] Fire on SceneItem save (label change) + - [ ] Debouncing if rapid edits cause webhook storm +- [ ] LanceDB schema: vector + source reference + chunk index + model name + model version + timestamp + +### Phase 2 — Initial backfill + +- [ ] Script to embed all existing PBS content from current state of MySQL +- [ ] Run once to populate LanceDB +- [ ] Document the script for future re-embedding when model changes + +### Phase 3 — `search_projects` MCP tool replacement + +- [ ] Update pbs-hub-mcp's `search_projects` tool to use LanceDB +- [ ] Internally: embed query with same model, vector similarity search, return top N +- [ ] Return shape matches v1 (project summaries with score) +- [ ] Integration test: known content returns expected matches +- [ ] Deploy to staging, validate against real searches +- [ ] Deploy to prod + +### Phase 4 — Polish and learning loop + +- [ ] Monitor real search behavior — which queries return what +- [ ] Adjust chunking thresholds based on observed content sizes +- [ ] Consider hybrid ranking if pure vector misses obvious title hits +- [ ] Re-evaluate embedding model based on real-world result quality + +--- + +## Notes + +### Why this is its own plan + +The vector search design conversation expanded beyond what would fit in `pbs-hub-data-import-and-mcp`'s Phase 4 placeholder. Key design choices (embedding model, generation location, chunking strategy) all interact and benefit from explicit documentation. Spinning it out preserves that thinking for future-you. + +### Scale check + +At PBS's actual scale (one video/week, ~50/year), title-only search remains effective for the foreseeable future. This plan stays parked until title search becomes a real pain point. Realistic trigger window: 18+ months of accumulated content, or whenever Travis or Jenny says "I can't find that video about the thing where she talked about substitutions." + +### Cross-references + +- **Parent plan:** `pbs-hub-data-import-and-mcp` — this is its Phase 4 extracted into a standalone plan +- **Dependencies:** + - `pbs-hub-scene-management` schema must be live (content to embed) + - `pbs-hub-data-import-and-mcp` Phase 2 must be live (MCP server to update) + - n8n on herbydev (already running) for the embedding worker + - LanceDB / vector store infra (new for this work) + +### Key design rules to preserve + +- **Never mix embedding models in one index.** Store model name + version with every vector. Migration = full re-embed. +- **Embedding generation is async, not request-blocking.** pbs-hub never waits on an embedding API call. +- **Chunking is the rare case at PBS scale, not the default.** Per-scene rollup as the standard unit; chunk only when content exceeds ~3500 tokens. +- **Eventual consistency is fine.** A few seconds of lag between content save and searchability is invisible. + +### Flagged for plan readers + +- This plan is **not on the immediate critical path.** Don't pull it forward without a real pain point. Premature build wastes time tuning a system against a too-small corpus. +- The embedding model is the *workhorse* of the system. If you ever need to change it, plan an afternoon for re-embedding the whole library. Trivial at PBS scale, but real work. +- LanceDB / sqlite-vec / wherever-the-ecosystem-lands at Phase 4 kickoff — don't lock in the choice from this plan without sanity-checking what's current at that point. + +### Conversation context + +This plan emerged from a chain of "while we're here..." extensions to the original scene-management work: +- Started as a Phase 4 stub in `pbs-hub-data-import-and-mcp` +- Travis flagged interest in embedded vector DBs (originally LanceDB) +- Conversation surfaced real architecture questions (embedding generation location, model choice, chunking strategy) +- Cost analysis showed hosted embeddings are cents at PBS scale — Travis chose hosted over local +- Travis recognized scale realities ("it's probably going to take me a while to generate enough video ideas") and asked to defer with documented direction + +That last beat is the right one. Plan filed, build deferred. \ No newline at end of file