wiki-vault/Wiki/Dev/ai-memory-architecture.md

type	path	tags	sources	created	updated
synthesis	Wiki/Dev	ob1 embeddings pgvector wiki automation mcp	Sources/Dev/ai-memory-architecture-research Sources/Dev/second-brain Sources/Homelab/ob1-main-deployment Sources/Homelab/ob1-deployment Sources/Dev/email-to-obsidian-automation Sources/Dev/email-obsidian-git-pipeline Sources/Homelab/homelab-mcp-server Sources/Homelab/2026-05-06-homelab-mcp-server-stand-up Sources/Dev/wiki-vault-migration Sources/Homelab/postgres-consolidation-ob1-reflection-layer Sources/Homelab/postgres-consolidation-followups Sources/Homelab/2026-05-15-postgres-consolidation-phase0-recon Sources/Homelab/2026-05-15-postgres-consolidation-phase2a-result Sources/Homelab/ob1-jsonl-watcher-ambient-capture-from-claude-code-sessions Sources/Homelab/vault-semantic-search Sources/Homelab/herbylab-project-management Sources/Homelab/trellis-mcp	2026-05-08	2026-05-24

AI Memory Architecture

Synthesis

Travis's AI memory system is not a single tool but a layered architecture: a structured wiki vault for human-readable knowledge, a pgvector semantic store for AI-queryable memory, a homelab MCP server to expose both to AI agents, and pipelines to keep all layers in sync. These components appear across 15+ projects and represent a coherent long-term investment in AI-assisted knowledge work.

Update (2026-05-24 compile). Two things in the original synthesis below are now superseded and corrected in-line: (1) OB1 was never "Supabase" — it's a self-hosted pgvector/pgvector:pg17 container, since consolidated into the petalbrain database; (2) the "n8n parallel write" ingest path for OB1 was replaced by a JSONL session watcher. The Consolidated Substrate and Trellis: the third leg sections below capture the May-2026 state; the older sections are kept for design-history continuity.

The Consolidated Substrate (petalbrain)

As of the May-2026 postgres consolidation, all three memory systems share one Postgres instance (homelab-postgres, pgvector:pg17) under one database, petalbrain, with schema-level isolation and a dedicated service role per system:

petalbrain (Postgres / pgvector:pg17, on Proxmox, Tailscale-only)
├── schema wiki       — owner vault_mcp    — wiki.artifacts / projects / notes
├── schema ob1        — owner ob1_mcp      — ob1.thoughts (raw) + ob1.reflections (compiled)
├── schema trellis    — owner trellis_mcp  — threads / events / tags
└── public.embeddings — shared vector store, all sources, 768-d nomic-embed-text
                        HNSW vector_cosine_ops (m=16, ef_construction=64)
                        512-token chunks / 64-token overlap

public.embeddings is the load-bearing idea: one table for every source, keyed by (source_schema, source_table, source_id, embedding_model), so "find similar to X across everything" is a single ORDER BY embedding <-> query rather than a UNION across per-system tables. Cross-schema reads are granted narrowly (ob1_mcp reads wiki + trellis for reflection synthesis; lovebug is RW everywhere). Isolation is enforced by Postgres at the role level, not by trusting MCP code to behave — matching the same "separate the who, not the where" pattern used for host users and Authentik.

The embedding model is local Ollama nomic-embed-text (768-d) — never a hosted provider for memory. Switching it is a deliberate full re-embed, which is why embedding_model is part of the uniqueness key.

OB1's own two layers

OB1 now mirrors the wiki's Sources/Wiki split internally: ob1.thoughts is the raw capture layer; ob1.reflections is a compiled layer written by a daily reflection job (claude-sonnet-4-6 via the Anthropic Messages API direct) that synthesizes generalizable lessons with citations back to source rows. Same discipline as the wiki — mixing raw and synthesized content in one table pollutes semantic neighbors. Ambient capture is the Sources/Homelab/ob1-jsonl-watcher-ambient-capture-from-claude-code-sessions: it tails Claude Code / Cowork session files and posts turn-pairs to OB1's capture_thought. "Save everything, filter later."

Trellis: the third leg

The original two-system framing (wiki + OB1) is now three: Sources/Homelab/trellis-mcp is resumable working state — named threads with checkpoints that survive conversation boundaries. The clean separation the stack settled on: CLAUDE.md = instructions (how to behave), Trellis = record (where we are in the doing), wiki + OB1 = lessons and outcomes (what we learned/built). None should swallow another's job. Trellis lives in petalbrain.trellis; its tool surface is being refined (Sources/Homelab/trellis-mcp-tool-surface-followups).

The Three Memory Layers

Layer 1: Structured Wiki (wiki-vault)
  - Human-readable Markdown
  - Obsidian wikilinks + navigation
  - Compiled by LLM (wiki-maintenance skill)
  - Git-versioned
  - Written by: MCP server, email pipeline, Travis/Jenny

Layer 2: Semantic Memory (OB1)   [see "Consolidated Substrate" above for current state]
  - self-hosted pgvector:pg17 (petalbrain), NOT Supabase
  - Embeddings of vault content + session context (Ollama nomic-embed-text, 768-d)
  - Queryable by AI agents via semantic search over public.embeddings
  - Written by: JSONL session watcher → capture_thought (NOT n8n parallel write)
  - Access: Tailscale-only

Layer 3: Agent Context (Dispatch / Lovebug)
  - Active working memory within Claude Code context windows
  - Pre-compact hook (Phase 7) will drain important context → OB1
  - MCP server provides tool access to query OB1 and wiki from active sessions

Ingest Paths

Multiple input channels feed into the wiki-vault and, in parallel, into OB1:

1. MCP server (mcp.herbylab.dev) — Claude Code and other AI agents write session notes and project updates directly to Sources/ via the MCP tools. This is the primary agentic write path.

2. Email pipeline — Gmail → n8n → Gitea push → vault. Designed for mobile/async capture: send an email, it becomes a session note. Built and validated in the email-obsidian-git-pipeline session.

3. Manual / direct — Travis or Jenny write directly to the vault via Obsidian.

4. Parallel OB1 write (planned) — n8n intercepts all vault writes and mirrors them to OB1 as embeddings. Not yet live; pending OB1 deployment Phase 4.

Query Paths

• Wiki — humans browse via Obsidian; AI agents query via MCP vault-read tools
• OB1 — AI agents query via MCP OB1-recall tools (semantic search over embeddings)
• Context injection — the homelab MCP server exposes both layers as Claude Code tools, enabling Lovebug to pull relevant wiki pages or OB1 memories as needed within a session

Harness vs. Model Architecture

The AI memory research session explored the design question: should AI memory be implemented at the model layer (fine-tuning, RLHF on personal data) or the harness layer (RAG, context injection, external retrieval)? The conclusion was harness-layer: model weights are slow and expensive to update, while a pgvector store + MCP injection is fast, reversible, and inspectable. Poolside and Pi.dev were evaluated as examples of model-layer approaches — useful for code but not for personal episodic memory.

The two memory modes are distinct:

• Episodic (session notes, project logs) → OB1 embeddings for temporal/event recall
• Semantic (wiki entity pages, synthesis) → wiki-vault for stable structured knowledge

The Second-Brain Pattern

The second-brain project captures the Karpathy LLM wiki pattern: pull raw content → transcribe → extract entities/facts → review → publish to wiki. This is exactly what the wiki-maintenance compile loop implements. The compile loop is the "review → publish" step; the MCP server + email pipeline are the "pull → capture" steps. OB1 adds the AI-queryable retrieval layer that the pure wiki-markdown model lacks.

MCP Server as the Integration Point

The homelab MCP server (mcp.herbylab.dev, deployed 2026-05-06) is the critical integration point that makes this architecture operational. Before MCP, each layer was separate — the wiki was browsed in Obsidian, OB1 was planned but not integrated, and AI agents had no way to query either during a session. The MCP server exposes vault tools and (once OB1 is live) OB1 recall as Claude Code tool calls, making the entire memory stack available to Lovebug mid-session.

Sources

ai-memory-architecture-research (session notes) provides the harness-vs-model architecture decision, the pgvector choice, and the Poolside/Pi.dev research. This is the foundational design session for OB1.

second-brain (project plan) captures the Karpathy LLM wiki pattern and maps it to the wiki-vault compile loop + MCP server design. Establishes the pull→transcribe→extract→review→publish pipeline.

ob1-main-deployment (project plan) provides the full OB1 deployment architecture: Supabase + pgvector, Phases 0–6, embedding pipeline design.

ob1-deployment (addendum) captures the locked decisions: Tailscale-only access, n8n parallel write pipeline, Phase 7 Dispatch pre-compact hook. Critical refinements to the main plan.

email-to-obsidian-automation and email-obsidian-git-pipeline document the Gmail → n8n → Gitea → vault ingest path. The pipeline is built and validated as of May 2026.

homelab-mcp-server and 2026-05-06-homelab-mcp-server-stand-up document the MCP server deployment: FastMCP Streamable HTTP, Authentik OAuth2, Cloudflare Tunnel. The integration point connecting all memory layers to AI agents.

wiki-vault-migration documents the vault consolidation from pbs-projects + homelab-projects into this unified structure, establishing the wiki-vault as the canonical structured knowledge store.

postgres-consolidation-ob1-reflection-layer (project plan) is the source for the single-database / three-schema / four-service-user model, the shared public.embeddings design, and OB1's two-layer captures+reflections split. 2026-05-15-postgres-consolidation-phase0-recon is the primary evidence that OB1 is pgvector:pg17 + Ollama nomic-embed-text (768-d), not Supabase — it inventories the live instance directly. phase2a-result confirms the schemas, roles, and public.embeddings table landed with verified permission isolation. postgres-consolidation-followups supplies the reflection-model choice (claude-sonnet-4-6 direct), chunking parameters (512/64), and the enricher loop bug.

ob1-jsonl-watcher (project plan) is the source for the ambient-capture-via-JSONL design that replaced the n8n parallel-write idea. trellis-mcp establishes the third memory leg and the instructions/record/lessons three-way separation. herbylab-project-management documents the Postgres-backed artifact model and the NOTIFY→n8n→Trello projection. vault-semantic-search specifies the (still-unbuilt) vault:search tool over the wiki schema.

Open questions

Several earlier open questions were resolved by the May-2026 consolidation: OB1's substrate is confirmed (pgvector:pg17 / petalbrain, not Supabase); ambient capture is the JSONL watcher, not n8n parallel-write (so there is no wiki↔OB1 dual-source-of-truth problem — vault is content, OB1 indexes session content); and the shared public.embeddings table is live.

Still open:

• Reflector quality — the daily reflection job's synthesis prompt is unproven; first non-dry-run output needs review before the 3am timer is trusted (postgres-consolidation-followups Phase 1).
• ob1-enricher PENDING_QUERY loop — must be fixed before the enricher is restarted, or the metadata backlog grows.
• Re-embed on compile — when the wiki compile loop edits a Wiki page, its public.embeddings row goes stale. There is still no re-embed step in the compile procedure. (Agent inference, not stated in any source — flagged.)
• vault:search not yet built — the semantic-search MCP tool over the wiki schema is designed (Sources/Homelab/vault-semantic-search) but unbuilt; until it ships, the wiki half of the substrate isn't actually queryable by agents.
• claude/competent-knuth-de9483 branch — artifact tooling that landed via a separate path; merge/abandon/cherry-pick still undecided.
• Dispatch pre-compact hook (Phase 7) — still depends on a Dispatch lifecycle event; the JSONL watcher covers Claude Code/Cowork capture in the meantime.