wiki-vault/Sources/Dev/pbs-hub-vector-search.md

created	path	project	tags	type
2026-05-19	Sources/Dev	pbs-hub-vector-search	pbs pbs-hub mcp vector-search embeddings lancedb n8n openai rag	project-plan

Goal

Add semantic vector search to pbs-hub-mcp's search_projects tool, replacing the v1 title-only implementation. Lets an LLM (or human via paste-import-style search UI later) find projects by meaning rather than by exact title match — "find me the cobbler video where Jenny talks about gluten substitution" instead of needing to remember the title.

This is the Phase 4 deferred work from pbs-hub-data-import-and-mcp. Pulled out into its own plan because the design conversation got substantial enough to warrant its own home, and because the feature isn't needed until PBS has accumulated enough content for title search to feel limiting. At one video/week that's probably 18+ months out.

Trigger to start build: title search on the real PBS project library has become a real friction point. Until then this plan sits as documented direction.

---

Locked Decisions (Preliminary)

These represent the leaning state from design discussion. To be confirmed at Phase 4 kickoff with whatever's true at that point in time (model availability, infra state, etc.).

Vector store

LanceDB is the leaning choice. Alternative considered: sqlite-vec. Both are embedded (no separate container needed), both fit comfortably at PBS scale. LanceDB's columnar storage, versioning, and modern AI/RAG ecosystem positioning give it the edge. Either is defensible.

Open at kickoff: confirm LanceDB hasn't been superseded by something better; confirm Python ecosystem maturity is still strong.

Embedding model

OpenAI text-embedding-3-small (or whatever its successor is at Phase 4 kickoff).

Reasons:

• Cost is negligible at PBS scale ($0.02/M tokens — entire library embedding is likely cents total)
• Mature, stable, multi-year API lifecycle
• Wide tooling support; every vector DB integrates natively
• Default 1536 dimensions is plenty for retrieval quality at this scale

Embedding model must remain consistent. All vectors in the index must be generated by the same model. Switching models = re-embedding everything. Cost of re-embedding is trivial; the operational discipline of "never silently mix models in the same index" is what matters.

Store model name + version alongside each embedding as a column in LanceDB. Makes mixed-model bugs impossible to introduce silently and makes migration trivially debuggable.

Embedding generation location

Async worker pattern, n8n-driven.

Architecture:

pbs-hub (pbs Linode) — content changes
       ↓ webhook
n8n (herbydev) — picks up event
       ↓ calls embedding API
OpenAI (or whichever provider)
       ↓ writes vector
LanceDB (likely on pbs Linode)
       ↑ queried by
pbs-hub-mcp (pbs Linode) — when LLM client searches

Why this shape:

• Embedding latency doesn't block user-facing writes in pbs-hub
• pbs-hub stays focused on CMS work; no embedding logic in the request path
• n8n is already the right hammer for "when X happens, do Y" — pattern is reusable
• Failure isolation: embedding gen failure doesn't lose user content
• Eventual consistency is fine (5-second lag between save and searchability is invisible)

Vector store physical location

LanceDB on the pbs Linode. Reasoning:

• pbs-hub-mcp queries it on every search → read latency matters
• Writes are async/infrequent (whenever content changes) → write latency tolerable
• Backups go alongside other pbs Linode data
• Cross-site reads (MCP on Linode reading from herbydev) would be slower than cross-site writes (n8n on herbydev writing to Linode)

n8n on herbydev writes across to the pbs Linode's LanceDB volume. pbs-hub-mcp reads locally on Linode.

Embedding units (what gets embedded)

Per-scene rollup as the primary unit, with per-project-field rollup for top-level fields.

• Project-level: one embedding for outline, one for treatment. Each metadata value embedded separately if it's substantive (logline yes, tagline probably not — decide at kickoff).
• Scene-level: one embedding per scene, content = concatenated purpose + in_shot + background + script + scene item labels.
• Chunking: only when a content unit exceeds ~3500 tokens (well under the 8191 ceiling of text-embedding-3-small). Overlap ~300 tokens between chunks when chunking is needed.

Reasoning:

• Matches mental model of search: "find the cobbler scene about substitution" → return a scene, not a paragraph
• At PBS scale, almost no scene rollup will exceed 3500 tokens. Chunking is the rare case, not the default
• Storing source reference + chunk index alongside each embedding means search results can name what they came from ("scene 3 of Blueberry Cobbler, chunk 0")
• Easy to add per-field precision later (Phase 5+) without throwing away the per-scene index

search_projects tool surface

Replaces the v1 title-only behavior. The MCP tool surface stays the same shape — query in, results out — but the body of the search uses vectors.

• Accepts same query parameter (free text)
• Returns same result shape (project summaries with relevance score)
• Internally: embed the query with the same model, compare against indexed vectors, return top N
• Optional: hybrid ranking (combine vector similarity with title BM25). Stretch goal at Phase 4 kickoff; pure vector is fine for v1.

---

Open Items (revisit at Phase 4 kickoff)

• Vector store final pick — LanceDB still the leading option at kickoff time, or has the ecosystem moved? Sanity-check sqlite-vec and any new entrants.
• Embedding model final pick — confirm text-embedding-3-small is still current, not deprecated; consider Voyage or Cohere if pricing or quality has shifted.
• Embedding generation: hosted vs local — at Phase 4 kickoff, reconsider hosted-vs-local. Local (Ollama + nomic-embed-text on herbydev) was discussed but hosted won on grounds of cost being negligible and operational simplicity. Worth a fresh look if homelab AI infra has matured.
• Tagline embedding — embed tagline as its own vector, or skip (too short to be useful)? Decide once real metadata patterns emerge.
• Hybrid ranking — pure vector vs BM25+vector hybrid. Pure vector is v1; hybrid is stretch goal or future enhancement.
• Dimensions config — text-embedding-3-small default is 1536; configurable down to 512 for storage savings. At PBS scale, default is fine. Reconsider if storage becomes a concern.
• LanceDB volume strategy — docker volume mount? Where does it live on the host? Backup story? Surface for Lovebug at kickoff.
• Re-embedding workflow — script or n8n flow to regenerate all embeddings when model changes or chunking strategy changes. Worth designing once Phase 4 starts.
• Search ranking transparency — return raw similarity scores to the LLM, or thresholded boolean ("yes this matched")? Affects how the LLM reasons about result confidence.

---

Phases

Phase 1 — Vector store + embedding pipeline foundation

• LanceDB deployed on pbs Linode (or alternative if reconsidered at kickoff)
  • Docker volume strategy decided and implemented
  • Backup approach defined
• n8n flow for "content change → embed → write to LanceDB"
  • Webhook receiver from pbs-hub
  • Embedding API call (OpenAI or alternative)
  • Write to LanceDB
  • Error handling + retry logic
• pbs-hub webhook firing on content changes
  • Fire on Project.outline / treatment / metadata save
  • Fire on Scene save (any briefing or script field)
  • Fire on SceneItem save (label change)
  • Debouncing if rapid edits cause webhook storm
• LanceDB schema: vector + source reference + chunk index + model name + model version + timestamp

Phase 2 — Initial backfill

• Script to embed all existing PBS content from current state of MySQL
• Run once to populate LanceDB
• Document the script for future re-embedding when model changes

Phase 3 — search_projects MCP tool replacement

• Update pbs-hub-mcp's search_projects tool to use LanceDB
• Internally: embed query with same model, vector similarity search, return top N
• Return shape matches v1 (project summaries with score)
• Integration test: known content returns expected matches
• Deploy to staging, validate against real searches
• Deploy to prod

Phase 4 — Polish and learning loop

• Monitor real search behavior — which queries return what
• Adjust chunking thresholds based on observed content sizes
• Consider hybrid ranking if pure vector misses obvious title hits
• Re-evaluate embedding model based on real-world result quality

---

Notes

Why this is its own plan

The vector search design conversation expanded beyond what would fit in pbs-hub-data-import-and-mcp's Phase 4 placeholder. Key design choices (embedding model, generation location, chunking strategy) all interact and benefit from explicit documentation. Spinning it out preserves that thinking for future-you.

Scale check

At PBS's actual scale (one video/week, ~50/year), title-only search remains effective for the foreseeable future. This plan stays parked until title search becomes a real pain point. Realistic trigger window: 18+ months of accumulated content, or whenever Travis or Jenny says "I can't find that video about the thing where she talked about substitutions."

Cross-references

• Parent plan: pbs-hub-data-import-and-mcp — this is its Phase 4 extracted into a standalone plan
• Dependencies:
  • pbs-hub-scene-management schema must be live (content to embed)
  • pbs-hub-data-import-and-mcp Phase 2 must be live (MCP server to update)
  • n8n on herbydev (already running) for the embedding worker
  • LanceDB / vector store infra (new for this work)

Key design rules to preserve

• Never mix embedding models in one index. Store model name + version with every vector. Migration = full re-embed.
• Embedding generation is async, not request-blocking. pbs-hub never waits on an embedding API call.
• Chunking is the rare case at PBS scale, not the default. Per-scene rollup as the standard unit; chunk only when content exceeds ~3500 tokens.
• Eventual consistency is fine. A few seconds of lag between content save and searchability is invisible.

Flagged for plan readers

• This plan is not on the immediate critical path. Don't pull it forward without a real pain point. Premature build wastes time tuning a system against a too-small corpus.
• The embedding model is the workhorse of the system. If you ever need to change it, plan an afternoon for re-embedding the whole library. Trivial at PBS scale, but real work.
• LanceDB / sqlite-vec / wherever-the-ecosystem-lands at Phase 4 kickoff — don't lock in the choice from this plan without sanity-checking what's current at that point.

Conversation context

This plan emerged from a chain of "while we're here..." extensions to the original scene-management work:

• Started as a Phase 4 stub in pbs-hub-data-import-and-mcp
• Travis flagged interest in embedded vector DBs (originally LanceDB)
• Conversation surfaced real architecture questions (embedding generation location, model choice, chunking strategy)
• Cost analysis showed hosted embeddings are cents at PBS scale — Travis chose hosted over local
• Travis recognized scale realities ("it's probably going to take me a while to generate enough video ideas") and asked to defer with documented direction

That last beat is the right one. Plan filed, build deferred.