wiki-vault/Sources/Dev/second-brain-search-and-synthesis.md

created	path	project	tags	type
2026-05-25	Sources/Dev	second-brain-search-and-synthesis	second-brain pgvector embeddings search lovebug mcp	project-plan

Second Brain — Search & Synthesis

Follow-on to second-brain. The base pipeline shipped (Postgres + pgvector, two-machine architecture, web app at brain.herbylab.dev, embeddings flowing). This plan covers what's next: a real search feature and the open question of what synthesis layer fits the brain — explicitly distinct from the wiki-LLM project's synthesis layer.

Goal

Phase 1: Ship a working search over second-brain extractions with progressively-richer retrieval modes exposed as a single dial (levels 1–4). Each level is a strict superset of the previous, so the dial is monotonic and the UX is "how hard do I want this to try?"

Phase 2: Design a synthesis layer that fits the second-brain's corpus shape (other people's content) without diluting the wiki-LLM's signal (Travis's own content). Scope deferred until corpus grows and use patterns emerge.

Locked Decisions

Architecture relationship to wiki-LLM

• Second-brain and wiki-LLM are separate systems with separate synthesis needs. The wiki-LLM compiles Travis's generated content (project plans, session notes) into entities/synthesis. The second-brain ingests other people's content (articles, videos). Mixing them would dilute the wiki's signal.
• Second-brain extractions do NOT flow into the wiki-vault's Sources/ — that was a considered option earlier in the planning conversation and rejected on noise-contamination grounds.
• Wiki-LLM's repeated-themes / pattern-finding output is interesting input for the brain's synthesis but is not a complete answer for what the brain itself needs.

Search architecture — the dial

Four levels, each adding to the previous. User picks level per-search via a dial in the UI.

• Level 1 — Semantic only. Query → nomic-embed-text embedding → pgvector nearest-neighbor over extraction summaries. Returns top N by cosine distance. What's already half-built; this is the baseline.
• Level 2 — Hybrid (semantic + lexical). Run Level 1 in parallel with Postgres full-text search (tsvector). Merge results via Reciprocal Rank Fusion (RRF), k=60 convention. Fixes the proper-noun problem (e.g., searching "RepoWise" should hit exact-term matches, not vibes-matches).
• Level 3 — Multi-field embeddings. Embed summary, key_points, and title separately (or as a combined search-blob). Search each. Improves recall when the surface text of the summary doesn't include the user's query terms but a key_point does.
• Level 4 — LLM rerank. Take the top ~10 results from Level 3 and rerank via a single Claude SDK call. Highest quality, slow (2–5s), and uses subscription tokens. No dedicated reranker model (rejected: adds infra dependency, marginal win over LLM rerank at this scale).

Search implementation decisions

• Dial UI, not checkboxes. Single control, four positions. Monotonic.
• Comparison view is a separate science-project page, not bolted onto the main search. Runs all four levels for one query and shows results side-by-side. Built later, after the dial is live and there's data to compare against.
• Query logging is non-negotiable. Log every query, dial level, result IDs returned, and (when added) click-throughs. Without logs, "does v3 help" is vibes; with logs, it's analyzable.
• No HNSW/IVFFlat index yet. Sequential scan is fine at current corpus size (<10 sources). Flagged as a v2.5 task — ~10 minutes of work when the corpus passes ~100 sources or query latency becomes felt.
• MCP-friendly contract. The search function is designed clean (input: query, level, filters; output: result list) so when the brain becomes MCP-exposed, it's a direct lift. No UI state baked into the search core.

Synthesis layer (Phase 2)

• No user story yet. Acknowledged honestly: current usage pattern is "see article, drop in chat, see what happens." Without a clear moment of use, designing synthesis features is guessing.
• Curation cuts through noise — whatever synthesis emerges, it has to be selective enough to be worth opening. Volume-of-content synthesis = noise = ignored.
• Phase 2 stays in design-thinking mode until either (a) the corpus grows enough that real use patterns emerge, or (b) MCP exposure creates an agent-side user (Lovebug) whose needs are concrete.

Open Items

Phase 1 — Search

• Result unit. Are results whole extractions, or chunks/snippets within them? Non-issue at Level 1 (one embedding per extraction), real call at Level 3 (multiple embeddings per source). Decide before Level 3 is wired.
• Highlighting / why-it-matched. Lexical hits can show matched terms easily; semantic hits can't. Decide whether to show both, neither, or lexical-only.
• Zero-results UX. Plain "no results" vs. "no results, here are the 3 closest with low scores" vs. "try removing filters" — pick one before shipping v1.
• Score visibility. Show similarity scores in the UI, or hide them? Travis is a power user; lean toward showing them, possibly behind a toggle.
• Latency budget per level. L1 <100ms, L2 likely <200ms, L3 similar, L4 2–5s. The dial UI needs to communicate "you've asked for a slow mode" — spinner with mode label, or similar.
• Result caching. Skip for L1–L3. For L4 (LLM rerank), cache (query, mode, filters) → results with a TTL. Implement when L4 lands.
• Pagination / result count. Top 10 hardcoded for v1; revisit when corpus and use patterns demand it.
• Default domain scope. Search globally across all domains by default, or remember a "current domain" context? Lean global; flag for review post-v1.
• Permission / visibility column. Not needed now. Easy to add a visibility column on extractions for future-proofing; harder to retrofit if personal-context content ever becomes shareable. Decide whether to add as a no-op now.

Phase 2 — Synthesis

Pinned for later. The conversation surfaced several candidate shapes but no commitment:

• "What's converging?" — patterns across many sources. Closest to what the wiki-LLM's pattern-finding output already produces. Plausible starting point.
• "What's worth my time?" — given current interests, surface relevant sources. Feed-shaped, not wiki-shaped.
• "What's new since I last looked?" — corpus diff for a regular review habit.
• "How does this relate to my thinking?" — bridge between brain and wiki. Most powerful, hardest to do well without coupling the systems.
• "What does the corpus think about X?" — topic-driven aggregation of source voices.

Decision deferred. Will revisit once corpus has >50 sources OR MCP integration lands and an agent-side use case becomes concrete.

Phases

Phase 1.1 — Level 1 (semantic baseline)

• Search box on dashboard
• Query embed via Ollama (nomic-embed-text)
• pgvector nearest-neighbor query against extraction summary embeddings
• Return top 10 with summary, URL, domain, status
• Domain + status filters as query params
• Query logging table + write path

Phase 1.2 — Level 2 (hybrid + RRF)

• Add tsvector column to extractions (generated column on summary + title + key_points text)
• Full-text search query path
• RRF merge (k=60) of semantic + lexical result lists
• Dial UI on search page — Level 1 / Level 2 selectable
• Update logging to capture dial level

Phase 1.3 — Level 3 (multi-field)

• Decide: separate embeddings per field, or one combined search-blob embedding
• Re-embed pass for existing extractions
• Update search query to consult all relevant embeddings
• Dial UI expanded to Level 3

Phase 1.4 — Level 4 (LLM rerank)

• Claude SDK rerank call on top ~10 results
• Prompt template for rerank (query + result summaries → reordered list)
• Latency-aware UI (spinner, "reranking…" label)
• Result cache (query, mode, filters) → results with TTL
• Dial UI expanded to Level 4

Phase 1.5 — Comparison view (science project)

• Separate route: run query at all four levels, render side-by-side
• Highlight rank deltas between modes
• Useful for tuning and for deciding when L4's cost is actually worth it

Phase 2 — Synthesis (deferred)

Held pending corpus growth, MCP exposure, or emergent use pattern.

Notes

Why the dial and not checkboxes

Earlier framing was four toggles, one per mode. The dial is monotonic — each level is a strict superset of the previous (L2 = L1 + lexical, L3 = L2 + multi-field, L4 = L3 + rerank). Users get one mental model ("how hard should this try") instead of four orthogonal options. UI is simpler. Lovebug's build is cleaner.

Why two synthesis systems

The wiki-LLM compiles Travis's own thinking — project plans, decisions, session notes. Its synthesis pages reflect his views and decisions. The second brain ingests other people's content. Feeding extractions into the wiki-LLM would mix "things I think" with "things I read," polluting the wiki. They share machinery (extract → store → compile) but the synthesis layers need to be separate. Phase 2 is the brain getting its own synthesis, scoped to its own corpus.

Phase 2 conversation summary

The synthesis question stalled on a real problem: there's no user story for the brain yet. Current pattern is reactive (interesting article → drop in chat → see what happens), not pull-based. Building synthesis features without a moment-of-use means guessing.

Three honest framings for how Phase 2 unblocks itself:

1. Push, not pull. A weekly digest / daily Obsidian note / Slack ping doesn't require the user to remember the brain exists. The system creates its own moment of use.
2. Future agent as primary user. When the brain becomes MCP-exposed, Lovebug becomes a consumer. "Should we use Traefik or Caddy" — answer pulls from brain. The synthesis that matters there is structured (claims with provenance), not prose.
3. Wait for the corpus to grow. Use the brain for a few weeks, see what's missing, design from data instead of speculation.

Travis's lean: synthesis must be curated to cut through noise. The wiki-LLM's repeated-themes output is interesting raw material but not a complete answer. Pinned for later.

Why query logging matters

The dial only earns its complexity if L2 measurably beats L1, L3 beats L2, etc. Without query logs, that comparison is vibes. With logs, you can replay queries across modes and see actual deltas. Cheap to build now, impossible to backfill — every un-logged query before logging lands is lost data.

Indexing strategy (flagged for later)

At <100 extractions, sequential scan on cosine distance is plenty fast. At ~1000, you'll feel it. Add an HNSW index when query latency crosses a felt threshold (~200ms for L1). IVFFlat is the alternative; HNSW is more common, slightly heavier on memory, generally better recall. The migration is ~10 minutes of work, not a project. Not now.

Open architectural question — should the brain become MCP soon?

Not in scope for this plan, but flagged: Phase 1's search core is being built to be MCP-friendly (clean input/output, no UI state in the search function itself). The moment that MCP exposure happens, the brain becomes useful to Lovebug as a knowledge source. That might itself be the user-story unlock for Phase 2 synthesis. Worth thinking about separately, but the design here doesn't preclude it — in fact it sets it up.