wiki-vault/Wiki/Dev/ai-memory-architecture.md

type	path	tags	sources	created	updated
synthesis	Wiki/Dev	ob1 embeddings pgvector wiki automation mcp	Sources/Dev/ai-memory-architecture-research Sources/Dev/second-brain Sources/Homelab/ob1-main-deployment Sources/Homelab/ob1-deployment Sources/Dev/email-to-obsidian-automation Sources/Dev/email-obsidian-git-pipeline Sources/Homelab/homelab-mcp-server Sources/Homelab/2026-05-06-homelab-mcp-server-stand-up Sources/Dev/wiki-vault-migration	2026-05-08	2026-05-08

AI Memory Architecture

Synthesis

Travis's AI memory system is not a single tool but a layered architecture: a structured wiki vault for human-readable knowledge, a pgvector semantic store for AI-queryable memory, a homelab MCP server to expose both to AI agents, and n8n pipelines to keep all layers in sync. These components appear across 8+ projects and represent a coherent long-term investment in AI-assisted knowledge work.

The Three Memory Layers

Layer 1: Structured Wiki (wiki-vault)
  - Human-readable Markdown
  - Obsidian wikilinks + navigation
  - Compiled by LLM (wiki-maintenance skill)
  - Git-versioned
  - Written by: MCP server, email pipeline, Travis/Jenny

Layer 2: Semantic Memory (OB1)
  - Supabase + pgvector
  - Embeddings of vault content + session context
  - Queryable by AI agents via semantic search
  - Written by: n8n parallel pipeline (mirrors vault writes)
  - Access: Tailscale-only

Layer 3: Agent Context (Dispatch / Lovebug)
  - Active working memory within Claude Code context windows
  - Pre-compact hook (Phase 7) will drain important context → OB1
  - MCP server provides tool access to query OB1 and wiki from active sessions

Ingest Paths

Multiple input channels feed into the wiki-vault and, in parallel, into OB1:

1. MCP server (mcp.herbylab.dev) — Claude Code and other AI agents write session notes and project updates directly to Sources/ via the MCP tools. This is the primary agentic write path.

2. Email pipeline — Gmail → n8n → Gitea push → vault. Designed for mobile/async capture: send an email, it becomes a session note. Built and validated in the email-obsidian-git-pipeline session.

3. Manual / direct — Travis or Jenny write directly to the vault via Obsidian.

4. Parallel OB1 write (planned) — n8n intercepts all vault writes and mirrors them to OB1 as embeddings. Not yet live; pending OB1 deployment Phase 4.

Query Paths

• Wiki — humans browse via Obsidian; AI agents query via MCP vault-read tools
• OB1 — AI agents query via MCP OB1-recall tools (semantic search over embeddings)
• Context injection — the homelab MCP server exposes both layers as Claude Code tools, enabling Lovebug to pull relevant wiki pages or OB1 memories as needed within a session

Harness vs. Model Architecture

The AI memory research session explored the design question: should AI memory be implemented at the model layer (fine-tuning, RLHF on personal data) or the harness layer (RAG, context injection, external retrieval)? The conclusion was harness-layer: model weights are slow and expensive to update, while a pgvector store + MCP injection is fast, reversible, and inspectable. Poolside and Pi.dev were evaluated as examples of model-layer approaches — useful for code but not for personal episodic memory.

The two memory modes are distinct:

• Episodic (session notes, project logs) → OB1 embeddings for temporal/event recall
• Semantic (wiki entity pages, synthesis) → wiki-vault for stable structured knowledge

The Second-Brain Pattern

The second-brain project captures the Karpathy LLM wiki pattern: pull raw content → transcribe → extract entities/facts → review → publish to wiki. This is exactly what the wiki-maintenance compile loop implements. The compile loop is the "review → publish" step; the MCP server + email pipeline are the "pull → capture" steps. OB1 adds the AI-queryable retrieval layer that the pure wiki-markdown model lacks.

MCP Server as the Integration Point

The homelab MCP server (mcp.herbylab.dev, deployed 2026-05-06) is the critical integration point that makes this architecture operational. Before MCP, each layer was separate — the wiki was browsed in Obsidian, OB1 was planned but not integrated, and AI agents had no way to query either during a session. The MCP server exposes vault tools and (once OB1 is live) OB1 recall as Claude Code tool calls, making the entire memory stack available to Lovebug mid-session.

Sources

ai-memory-architecture-research (session notes) provides the harness-vs-model architecture decision, the pgvector choice, and the Poolside/Pi.dev research. This is the foundational design session for OB1.

second-brain (project plan) captures the Karpathy LLM wiki pattern and maps it to the wiki-vault compile loop + MCP server design. Establishes the pull→transcribe→extract→review→publish pipeline.

ob1-main-deployment (project plan) provides the full OB1 deployment architecture: Supabase + pgvector, Phases 0–6, embedding pipeline design.

ob1-deployment (addendum) captures the locked decisions: Tailscale-only access, n8n parallel write pipeline, Phase 7 Dispatch pre-compact hook. Critical refinements to the main plan.

email-to-obsidian-automation and email-obsidian-git-pipeline document the Gmail → n8n → Gitea → vault ingest path. The pipeline is built and validated as of May 2026.

homelab-mcp-server and 2026-05-06-homelab-mcp-server-stand-up document the MCP server deployment: FastMCP Streamable HTTP, Authentik OAuth2, Cloudflare Tunnel. The integration point connecting all memory layers to AI agents.

wiki-vault-migration documents the vault consolidation from pbs-projects + homelab-projects into this unified structure, establishing the wiki-vault as the canonical structured knowledge store.

Open questions

• OB1 Phases 0–6 deployment timeline: what is the current blocker? The addendum was written but the deployment plan doesn't indicate Phase 0 is complete.
• When the n8n parallel write pipeline goes live, will it backfill existing vault content (all 62+ notes) into OB1, or only new writes going forward?
• The Dispatch pre-compact hook (Phase 7) depends on Dispatch exposing a lifecycle event. Is this a Dispatch feature that already exists, or does it require Dispatch development work?
• How will the MCP server's OB1 recall tool handle stale embeddings when vault content is updated by the compile loop? Will there be a re-embed step in the compile procedure?
• The wiki + OB1 parallel write model creates two sources of truth for the same content. Reconciliation strategy (wiki as authoritative; OB1 as derived index) should be documented once the pipeline goes live.