--- created: '2026-05-19' path: Sources/Dev project: pbs-hub-vector-search tags: - pbs - pbs-hub - mcp - vector-search - embeddings - lancedb - n8n - openai - rag type: project-plan --- ## Goal Add semantic vector search to `pbs-hub-mcp`'s `search_projects` tool, replacing the v1 title-only implementation. Lets an LLM (or human via paste-import-style search UI later) find projects by *meaning* rather than by exact title match — "find me the cobbler video where Jenny talks about gluten substitution" instead of needing to remember the title. This is the Phase 4 deferred work from `pbs-hub-data-import-and-mcp`. Pulled out into its own plan because the design conversation got substantial enough to warrant its own home, *and* because the feature isn't needed until PBS has accumulated enough content for title search to feel limiting. At one video/week that's probably 18+ months out. **Trigger to start build:** title search on the real PBS project library has become a real friction point. Until then this plan sits as documented direction. --- ## Locked Decisions (Preliminary) These represent the leaning state from design discussion. To be confirmed at Phase 4 kickoff with whatever's true at that point in time (model availability, infra state, etc.). ### Vector store **LanceDB** is the leaning choice. Alternative considered: `sqlite-vec`. Both are embedded (no separate container needed), both fit comfortably at PBS scale. LanceDB's columnar storage, versioning, and modern AI/RAG ecosystem positioning give it the edge. Either is defensible. Open at kickoff: confirm LanceDB hasn't been superseded by something better; confirm Python ecosystem maturity is still strong. ### Embedding model **OpenAI `text-embedding-3-small`** (or whatever its successor is at Phase 4 kickoff). Reasons: - Cost is negligible at PBS scale ($0.02/M tokens — entire library embedding is likely cents total) - Mature, stable, multi-year API lifecycle - Wide tooling support; every vector DB integrates natively - Default 1536 dimensions is plenty for retrieval quality at this scale **Embedding model must remain consistent.** All vectors in the index must be generated by the same model. Switching models = re-embedding everything. Cost of re-embedding is trivial; the *operational discipline* of "never silently mix models in the same index" is what matters. **Store model name + version alongside each embedding** as a column in LanceDB. Makes mixed-model bugs impossible to introduce silently and makes migration trivially debuggable. ### Embedding generation location **Async worker pattern, n8n-driven.** Architecture: ``` pbs-hub (pbs Linode) — content changes ↓ webhook n8n (herbydev) — picks up event ↓ calls embedding API OpenAI (or whichever provider) ↓ writes vector LanceDB (likely on pbs Linode) ↑ queried by pbs-hub-mcp (pbs Linode) — when LLM client searches ``` Why this shape: - Embedding latency doesn't block user-facing writes in pbs-hub - pbs-hub stays focused on CMS work; no embedding logic in the request path - n8n is already the right hammer for "when X happens, do Y" — pattern is reusable - Failure isolation: embedding gen failure doesn't lose user content - Eventual consistency is fine (5-second lag between save and searchability is invisible) ### Vector store physical location **LanceDB on the pbs Linode.** Reasoning: - pbs-hub-mcp queries it on every search → read latency matters - Writes are async/infrequent (whenever content changes) → write latency tolerable - Backups go alongside other pbs Linode data - Cross-site reads (MCP on Linode reading from herbydev) would be slower than cross-site writes (n8n on herbydev writing to Linode) n8n on herbydev writes across to the pbs Linode's LanceDB volume. pbs-hub-mcp reads locally on Linode. ### Embedding units (what gets embedded) **Per-scene rollup as the primary unit**, with per-project-field rollup for top-level fields. - **Project-level**: one embedding for `outline`, one for `treatment`. Each metadata value embedded separately if it's substantive (logline yes, tagline probably not — decide at kickoff). - **Scene-level**: one embedding per scene, content = concatenated `purpose` + `in_shot` + `background` + `script` + scene item labels. - **Chunking**: only when a content unit exceeds ~3500 tokens (well under the 8191 ceiling of `text-embedding-3-small`). Overlap ~300 tokens between chunks when chunking is needed. Reasoning: - Matches mental model of search: "find the cobbler scene about substitution" → return a scene, not a paragraph - At PBS scale, almost no scene rollup will exceed 3500 tokens. Chunking is the rare case, not the default - Storing source reference + chunk index alongside each embedding means search results can name what they came from ("scene 3 of Blueberry Cobbler, chunk 0") - Easy to add per-field precision later (Phase 5+) without throwing away the per-scene index ### `search_projects` tool surface Replaces the v1 title-only behavior. The MCP tool surface stays the same shape — query in, results out — but the *body* of the search uses vectors. - Accepts same `query` parameter (free text) - Returns same result shape (project summaries with relevance score) - Internally: embed the query with the same model, compare against indexed vectors, return top N - Optional: hybrid ranking (combine vector similarity with title BM25). Stretch goal at Phase 4 kickoff; pure vector is fine for v1. --- ## Open Items (revisit at Phase 4 kickoff) - [ ] **Vector store final pick** — LanceDB still the leading option at kickoff time, or has the ecosystem moved? Sanity-check `sqlite-vec` and any new entrants. - [ ] **Embedding model final pick** — confirm `text-embedding-3-small` is still current, not deprecated; consider Voyage or Cohere if pricing or quality has shifted. - [ ] **Embedding generation: hosted vs local** — at Phase 4 kickoff, reconsider hosted-vs-local. Local (Ollama + nomic-embed-text on herbydev) was discussed but hosted won on grounds of cost being negligible and operational simplicity. Worth a fresh look if homelab AI infra has matured. - [ ] **Tagline embedding** — embed tagline as its own vector, or skip (too short to be useful)? Decide once real metadata patterns emerge. - [ ] **Hybrid ranking** — pure vector vs BM25+vector hybrid. Pure vector is v1; hybrid is stretch goal or future enhancement. - [ ] **Dimensions config** — `text-embedding-3-small` default is 1536; configurable down to 512 for storage savings. At PBS scale, default is fine. Reconsider if storage becomes a concern. - [ ] **LanceDB volume strategy** — docker volume mount? Where does it live on the host? Backup story? Surface for Lovebug at kickoff. - [ ] **Re-embedding workflow** — script or n8n flow to regenerate all embeddings when model changes or chunking strategy changes. Worth designing once Phase 4 starts. - [ ] **Search ranking transparency** — return raw similarity scores to the LLM, or thresholded boolean ("yes this matched")? Affects how the LLM reasons about result confidence. --- ## Phases ### Phase 1 — Vector store + embedding pipeline foundation - [ ] LanceDB deployed on pbs Linode (or alternative if reconsidered at kickoff) - [ ] Docker volume strategy decided and implemented - [ ] Backup approach defined - [ ] n8n flow for "content change → embed → write to LanceDB" - [ ] Webhook receiver from pbs-hub - [ ] Embedding API call (OpenAI or alternative) - [ ] Write to LanceDB - [ ] Error handling + retry logic - [ ] pbs-hub webhook firing on content changes - [ ] Fire on Project.outline / treatment / metadata save - [ ] Fire on Scene save (any briefing or script field) - [ ] Fire on SceneItem save (label change) - [ ] Debouncing if rapid edits cause webhook storm - [ ] LanceDB schema: vector + source reference + chunk index + model name + model version + timestamp ### Phase 2 — Initial backfill - [ ] Script to embed all existing PBS content from current state of MySQL - [ ] Run once to populate LanceDB - [ ] Document the script for future re-embedding when model changes ### Phase 3 — `search_projects` MCP tool replacement - [ ] Update pbs-hub-mcp's `search_projects` tool to use LanceDB - [ ] Internally: embed query with same model, vector similarity search, return top N - [ ] Return shape matches v1 (project summaries with score) - [ ] Integration test: known content returns expected matches - [ ] Deploy to staging, validate against real searches - [ ] Deploy to prod ### Phase 4 — Polish and learning loop - [ ] Monitor real search behavior — which queries return what - [ ] Adjust chunking thresholds based on observed content sizes - [ ] Consider hybrid ranking if pure vector misses obvious title hits - [ ] Re-evaluate embedding model based on real-world result quality --- ## Notes ### Why this is its own plan The vector search design conversation expanded beyond what would fit in `pbs-hub-data-import-and-mcp`'s Phase 4 placeholder. Key design choices (embedding model, generation location, chunking strategy) all interact and benefit from explicit documentation. Spinning it out preserves that thinking for future-you. ### Scale check At PBS's actual scale (one video/week, ~50/year), title-only search remains effective for the foreseeable future. This plan stays parked until title search becomes a real pain point. Realistic trigger window: 18+ months of accumulated content, or whenever Travis or Jenny says "I can't find that video about the thing where she talked about substitutions." ### Cross-references - **Parent plan:** `pbs-hub-data-import-and-mcp` — this is its Phase 4 extracted into a standalone plan - **Dependencies:** - `pbs-hub-scene-management` schema must be live (content to embed) - `pbs-hub-data-import-and-mcp` Phase 2 must be live (MCP server to update) - n8n on herbydev (already running) for the embedding worker - LanceDB / vector store infra (new for this work) ### Key design rules to preserve - **Never mix embedding models in one index.** Store model name + version with every vector. Migration = full re-embed. - **Embedding generation is async, not request-blocking.** pbs-hub never waits on an embedding API call. - **Chunking is the rare case at PBS scale, not the default.** Per-scene rollup as the standard unit; chunk only when content exceeds ~3500 tokens. - **Eventual consistency is fine.** A few seconds of lag between content save and searchability is invisible. ### Flagged for plan readers - This plan is **not on the immediate critical path.** Don't pull it forward without a real pain point. Premature build wastes time tuning a system against a too-small corpus. - The embedding model is the *workhorse* of the system. If you ever need to change it, plan an afternoon for re-embedding the whole library. Trivial at PBS scale, but real work. - LanceDB / sqlite-vec / wherever-the-ecosystem-lands at Phase 4 kickoff — don't lock in the choice from this plan without sanity-checking what's current at that point. ### Conversation context This plan emerged from a chain of "while we're here..." extensions to the original scene-management work: - Started as a Phase 4 stub in `pbs-hub-data-import-and-mcp` - Travis flagged interest in embedded vector DBs (originally LanceDB) - Conversation surfaced real architecture questions (embedding generation location, model choice, chunking strategy) - Cost analysis showed hosted embeddings are cents at PBS scale — Travis chose hosted over local - Travis recognized scale realities ("it's probably going to take me a while to generate enough video ideas") and asked to defer with documented direction That last beat is the right one. Plan filed, build deferred.