diff --git a/Sources/Dev/index.md b/Sources/Dev/index.md index 88213cf..73e4d05 100644 --- a/Sources/Dev/index.md +++ b/Sources/Dev/index.md @@ -3,7 +3,7 @@ type: topic-landing path: Sources/Dev tags: - landing -updated: 2026-05-08 +updated: 2026-05-24 --- # Dev — Sources @@ -12,6 +12,13 @@ Project plans and session notes for coding, AI/ML, tooling, and dev environment ## Active Projects +- [[Sources/Dev/pbs-hub-scene-management|pbs-hub-scene-management]] — pre-production extension: outline/treatment/scenes + tablet Present Mode (8 phases) +- [[Sources/Dev/pbs-hub-pre-production-extension|pbs-hub-pre-production-extension]] — companion scope doc for the scene-management rail/canvas UI +- [[Sources/Dev/pbs-hub-data-import-and-mcp|pbs-hub-data-import-and-mcp]] — JSON paste-import + standalone `pbs-hub-mcp` server (4 phases) +- [[Sources/Dev/pbs-hub-vector-search|pbs-hub-vector-search]] — semantic search for `pbs-hub-mcp` (LanceDB + OpenAI embeddings; deferred ~18mo) +- [[Sources/Dev/zero-check-refactor|zero-check-refactor]] — manifest-driven routing + inconclusive guard for the validation gauntlet +- [[Sources/Dev/repo-hygiene-remediation|repo-hygiene-remediation]] — remediate the 2026-05 doc-audit findings across 23 repos + recurring review +- [[Sources/Dev/ml-drills-curriculum|ml-drills-curriculum]] — internalize the ML inner-loop via narrated drills (not project delivery) - [[Sources/Dev/instagram-automation-content-hub-plan|instagram-automation-content-hub-plan]] — master plan for PBS Instagram automation (Phases 1–5) - [[Sources/Dev/content-hub-phase5-architecture|content-hub-phase5-architecture]] — merge pbs-api + Content Hub into Blueprint Flask app - [[Sources/Dev/content-hub-phase5-planning|content-hub-phase5-planning]] — locked decisions for Content Hub Phase 5 @@ -42,11 +49,13 @@ Project plans and session notes for coding, AI/ML, tooling, and dev environment ## Recent Sessions +- [[Sources/Dev/2026-05-16-vault-integration-test|2026-05-16-vault-integration-test]] — vault indexing pipeline single-chunk verification *(test artifact — see flags)* +- [[Sources/Dev/2026-05-16-vault-integration-test-multi|2026-05-16-vault-integration-test-multi]] — multi-chunk indexing verification *(test artifact — see flags)* +- [[Sources/Dev/2026-05-15-agentic-validation-article-review|2026-05-15-agentic-validation-article-review]] — GitHub agentic-validation article; dominator analysis vs zero-check +- [[Sources/Dev/2026-05-10-dispatch-replacement-architecture-and-anthropic-billing-reality|2026-05-10-dispatch-replacement-architecture-and-anthropic-billing-reality]] — two-layer dispatch/agent split; subscription-vs-API billing - [[Sources/Dev/ai-memory-architecture-research|ai-memory-architecture-research]] — OB1 architecture; harness vs model layers; pgvector - [[Sources/Dev/cloudflare-cache-rules|cloudflare-cache-rules]] — last-match-wins rule ordering; cookie bypass fix - [[Sources/Dev/email-obsidian-git-pipeline|email-obsidian-git-pipeline]] — Gmail → n8n → Gitea → vault pipeline built -- [[Sources/Dev/instagram-automation-session-notes|instagram-automation-session-notes]] — MySQL schema; "Great Lockout of 2026" workaround -- [[Sources/Dev/pbs-api-db-connection-fix|pbs-api-db-connection-fix]] — pool_pre_ping=True, pool_recycle=1800 ## Completed Projects diff --git a/Sources/Homelab/index.md b/Sources/Homelab/index.md index dfc8a5f..369ee6c 100644 --- a/Sources/Homelab/index.md +++ b/Sources/Homelab/index.md @@ -3,7 +3,7 @@ type: topic-landing path: Sources/Homelab tags: - landing -updated: 2026-05-08 +updated: 2026-05-24 --- # Homelab — Sources @@ -12,6 +12,14 @@ Project plans and session notes for infrastructure, networking, hardware, and de ## Active Projects +- [[Sources/Homelab/postgres-consolidation-ob1-reflection-layer|postgres-consolidation-ob1-reflection-layer]] — consolidate wiki/ob1/trellis into `petalbrain` (3 schemas, 4 service users) + OB1 reflection layer +- [[Sources/Homelab/postgres-consolidation-followups|postgres-consolidation-followups]] — close-out tasks from the consolidation: reflector activation, verification, cleanup +- [[Sources/Homelab/trellis-mcp|trellis-mcp]] — MCP server for resumable threads of work across conversation boundaries +- [[Sources/Homelab/trellis-mcp-tool-surface-followups|trellis-mcp-tool-surface-followups]] — `pause_thread`→`checkpoint_thread` rename + optional `id` addressing +- [[Sources/Homelab/vault-semantic-search|vault-semantic-search]] — `vault:search` MCP tool over `public.embeddings` (wiki schema) +- [[Sources/Homelab/ob1-jsonl-watcher|ob1-jsonl-watcher]] — ambient capture of Claude Code/Cowork sessions into OB1 via JSONL tailing +- [[Sources/Homelab/herbylab-project-management|herbylab-project-management]] — projects/notes as Postgres-backed artifacts; Trello projection via NOTIFY→n8n +- [[Sources/Homelab/gitea-migration-to-pve|gitea-migration-to-pve]] — relocate Gitea to PVE LXC; two-instance cross-mirror with PBS - [[Sources/Homelab/homelab-mcp-server|homelab-mcp-server]] — MCP server (FastMCP + Authentik + Cloudflare Tunnel); Phase 1 tools - [[Sources/Homelab/ob1-main-deployment|ob1-main-deployment]] — OB1 AI memory system (Supabase + pgvector); Phases 0–6 - [[Sources/Homelab/ob1-deployment|ob1-deployment]] — OB1 addendum: Tailscale-only, parallel n8n write, Phase 7 Dispatch hook @@ -29,15 +37,19 @@ Project plans and session notes for infrastructure, networking, hardware, and de ## Recent Sessions -- [[Sources/Homelab/2026-05-06-homelab-mcp-server-stand-up|2026-05-06-homelab-mcp-server-stand-up]] — first MCP server deployment; Authentik OAuth + Cloudflare Tunnel; `mcp.herbylab.dev` -- [[Sources/Homelab/2026-05-08-homelab-vlan-renumber-switch-chip-to-bridge-alignment|2026-05-08-homelab-vlan-renumber-switch-chip-to-bridge-alignment]] — VLAN chip→bridge VID alignment (VID 11 lab, VID 21 private) -- [[Sources/Homelab/dns-knot-pihole-end-to-end|dns-knot-pihole-end-to-end]] — Knot DNS + Pi-hole chain live; rebind protection fix; EDE 15 debugging -- [[Sources/Homelab/homelab-dns-decision|homelab-dns-decision]] — locked Knot DNS architecture; split-horizon `.herbylab.dev` -- [[Sources/Homelab/server-stability-and-security-hardening|server-stability-and-security-hardening]] — MySQL OOM fix; WordPress cap; Cloudflare Bot Fight Mode; Traefik v3.6.11 -- [[Sources/Homelab/server-stability-mysql-oom|server-stability-mysql-oom]] — MySQL OOM investigation (earlier session) +- [[Sources/Homelab/2026-05-24-davinci-resolve-studio-install|2026-05-24-davinci-resolve-studio-install]] — Resolve Studio 20.3.3 on `herbyeos` via sheridans PKGBUILD +- [[Sources/Homelab/2026-05-22-pbs-nas-off-site-backup-via-rsync|2026-05-22-pbs-nas-off-site-backup-via-rsync]] — PVE→PBS LXC→NAS rsync off-site backup; day-1 restore validated +- [[Sources/Homelab/2026-05-21-traefik-routes-for-pihole-openwrt-us-test-authy-static-ip-cutove|2026-05-21-traefik-routes-for-pihole-openwrt...]] — pihole/openwrt behind Traefik; us-test-authy static-IP recovery +- [[Sources/Homelab/2026-05-18-claude-desktop-subprocess-leak-memory-management|2026-05-18-claude-desktop-subprocess-leak-memory-management]] — Dispatch subprocess zombies eating ~10.6GB RAM +- [[Sources/Homelab/2026-05-18-acl-shared-repo-permissions|2026-05-18-acl-shared-repo-permissions]] — POSIX default ACLs for herbyadmin/travadmin shared git tree +- [[Sources/Homelab/2026-05-15-postgres-consolidation-phase2a-result|2026-05-15-postgres-consolidation-phase2a-result]] — schemas + service users + `public.embeddings` landed disruption-free +- [[Sources/Homelab/2026-05-15-postgres-consolidation-phase0-recon|2026-05-15-postgres-consolidation-phase0-recon]] — pre-consolidation instance snapshot; OB1 is pg17/Ollama, not Supabase +- [[Sources/Homelab/2026-05-11-traefik-phase1-phase2-wildcard-canary|2026-05-11-traefik-phase1-phase2-wildcard-canary]] — Traefik Phase 1.6 wildcard + Phase 2 canary complete +- [[Sources/Homelab/2026-05-06-homelab-mcp-server-stand-up|2026-05-06-homelab-mcp-server-stand-up]] — first MCP server deployment; `mcp.herbylab.dev` ## Completed Projects +- [[Sources/Homelab/davinci-resolve-studio-install|davinci-resolve-studio-install]] — Resolve Studio 20.3.3 install on Endeavour tower ✓ - [[Sources/Homelab/ufw-docker-outage-fix|ufw-docker-outage-fix]] — UFW after.rules DOCKER-USER outbound rules fix ✓ - [[Sources/Homelab/wp-cron-supercronic-deploy|wp-cron-supercronic-deploy]] — WP-Cron disabled; Supercronic sidecar; Traefik block-wpcron ✓ - [[Sources/Homelab/2026-05-08-homelab-vlan-renumber-switch-chip-to-bridge-alignment|2026-05-08-homelab-vlan-renumber-switch-chip-to-bridge-alignment]] — VLAN renumber ✓ diff --git a/Sources/index.md b/Sources/index.md index a88ab11..a97408c 100644 --- a/Sources/index.md +++ b/Sources/index.md @@ -3,31 +3,31 @@ type: topic-landing path: Sources tags: - landing -updated: 2026-05-08 +updated: 2026-05-24 --- # Sources -All project plans and session notes. 62 files across Dev, Venture, and Homelab domains. +All project plans and session notes. 96 files across Dev, Venture, and Homelab domains. ## By Domain -- [[Sources/Dev/index|Dev]] — 27 projects, 5 sessions — coding, AI/ML, tooling, PBS tech +- [[Sources/Dev/index|Dev]] — 34 projects, 9 sessions — coding, AI/ML, tooling, PBS tech - [[Sources/Venture/index|Venture]] — 0 projects, 2 sessions — PBS business, content, brand -- [[Sources/Homelab/index|Homelab]] — 14 projects, 8 sessions — infrastructure, networking, deployments +- [[Sources/Homelab/index|Homelab]] — 23 projects, 22 sessions — infrastructure, networking, deployments - [[Sources/Reference/index|Reference]] — (empty) cross-domain reference notes ## Active Projects (cross-domain highlights) -- [[Sources/Homelab/ob1-main-deployment|ob1-main-deployment]] — OB1 AI memory (Supabase + pgvector, Phases 0–6) -- [[Sources/Dev/instagram-automation-content-hub-plan|instagram-automation-content-hub-plan]] — PBS Instagram automation (Phases 1–5) -- [[Sources/Dev/pbs-membership-loe1-phase3|pbs-membership-loe1-phase3]] — The Garden membership area -- [[Sources/Homelab/homelab-mcp-server|homelab-mcp-server]] — MCP server for AI tool access -- [[Sources/Dev/second-brain|second-brain]] — LLM wiki + AI memory architecture +- [[Sources/Homelab/postgres-consolidation-ob1-reflection-layer|postgres-consolidation-ob1-reflection-layer]] — `petalbrain` consolidation + OB1 reflection layer (largely landed) +- [[Sources/Dev/pbs-hub-scene-management|pbs-hub-scene-management]] — pbs-hub pre-production: scenes, scripts, tablet Present Mode +- [[Sources/Homelab/trellis-mcp|trellis-mcp]] — resumable work threads across conversation boundaries +- [[Sources/Dev/zero-check-refactor|zero-check-refactor]] — manifest-driven validation gauntlet + inconclusive guard +- [[Sources/Homelab/gitea-migration-to-pve|gitea-migration-to-pve]] — Gitea → PVE LXC, cross-mirrored with PBS ## Recent Sessions (cross-domain highlights) -- [[Sources/Homelab/2026-05-06-homelab-mcp-server-stand-up|2026-05-06-homelab-mcp-server-stand-up]] — MCP server live at `mcp.herbylab.dev` -- [[Sources/Homelab/2026-05-08-homelab-vlan-renumber-switch-chip-to-bridge-alignment|2026-05-08-homelab-vlan-renumber-switch-chip-to-bridge-alignment]] — VLAN renumber complete -- [[Sources/Dev/ai-memory-architecture-research|ai-memory-architecture-research]] — OB1 architecture research -- [[Sources/Dev/email-obsidian-git-pipeline|email-obsidian-git-pipeline]] — email-to-vault pipeline live +- [[Sources/Homelab/2026-05-22-pbs-nas-off-site-backup-via-rsync|2026-05-22-pbs-nas-off-site-backup-via-rsync]] — off-site PBS backup, restore validated day-1 +- [[Sources/Homelab/2026-05-15-postgres-consolidation-phase2a-result|2026-05-15-postgres-consolidation-phase2a-result]] — schemas + service users + shared embeddings table +- [[Sources/Dev/2026-05-10-dispatch-replacement-architecture-and-anthropic-billing-reality|2026-05-10-dispatch-replacement...]] — two-layer dispatch/agent architecture +- [[Sources/Homelab/2026-05-21-traefik-routes-for-pihole-openwrt-us-test-authy-static-ip-cutove|2026-05-21-traefik-routes...]] — more services behind Traefik; VM recovery diff --git a/Wiki/Dev/ai-memory-architecture.md b/Wiki/Dev/ai-memory-architecture.md index 4f3cb0b..a8e7047 100644 --- a/Wiki/Dev/ai-memory-architecture.md +++ b/Wiki/Dev/ai-memory-architecture.md @@ -18,15 +18,51 @@ sources: - "[[Sources/Homelab/homelab-mcp-server]]" - "[[Sources/Homelab/2026-05-06-homelab-mcp-server-stand-up]]" - "[[Sources/Dev/wiki-vault-migration]]" + - "[[Sources/Homelab/postgres-consolidation-ob1-reflection-layer]]" + - "[[Sources/Homelab/postgres-consolidation-followups]]" + - "[[Sources/Homelab/2026-05-15-postgres-consolidation-phase0-recon]]" + - "[[Sources/Homelab/2026-05-15-postgres-consolidation-phase2a-result]]" + - "[[Sources/Homelab/ob1-jsonl-watcher]]" + - "[[Sources/Homelab/vault-semantic-search]]" + - "[[Sources/Homelab/herbylab-project-management]]" + - "[[Sources/Homelab/trellis-mcp]]" created: 2026-05-08 -updated: 2026-05-08 +updated: 2026-05-24 --- # AI Memory Architecture ## Synthesis -Travis's AI memory system is not a single tool but a layered architecture: a structured wiki vault for human-readable knowledge, a pgvector semantic store for AI-queryable memory, a homelab MCP server to expose both to AI agents, and n8n pipelines to keep all layers in sync. These components appear across 8+ projects and represent a coherent long-term investment in AI-assisted knowledge work. +Travis's AI memory system is not a single tool but a layered architecture: a structured wiki vault for human-readable knowledge, a pgvector semantic store for AI-queryable memory, a homelab MCP server to expose both to AI agents, and pipelines to keep all layers in sync. These components appear across 15+ projects and represent a coherent long-term investment in AI-assisted knowledge work. + +> **Update (2026-05-24 compile).** Two things in the original synthesis below are now superseded and corrected in-line: (1) OB1 was never "Supabase" — it's a self-hosted `pgvector/pgvector:pg17` container, since consolidated into the `petalbrain` database; (2) the "n8n parallel write" ingest path for OB1 was replaced by a JSONL session watcher. The **Consolidated Substrate** and **Trellis: the third leg** sections below capture the May-2026 state; the older sections are kept for design-history continuity. + +### The Consolidated Substrate (`petalbrain`) + +As of the May-2026 postgres consolidation, all three memory systems share one Postgres instance (`homelab-postgres`, pgvector:pg17) under one database, **`petalbrain`**, with schema-level isolation and a dedicated service role per system: + +``` +petalbrain (Postgres / pgvector:pg17, on Proxmox, Tailscale-only) +├── schema wiki — owner vault_mcp — wiki.artifacts / projects / notes +├── schema ob1 — owner ob1_mcp — ob1.thoughts (raw) + ob1.reflections (compiled) +├── schema trellis — owner trellis_mcp — threads / events / tags +└── public.embeddings — shared vector store, all sources, 768-d nomic-embed-text + HNSW vector_cosine_ops (m=16, ef_construction=64) + 512-token chunks / 64-token overlap +``` + +`public.embeddings` is the load-bearing idea: one table for every source, keyed by `(source_schema, source_table, source_id, embedding_model)`, so "find similar to X across everything" is a single `ORDER BY embedding <-> query` rather than a UNION across per-system tables. Cross-schema reads are granted narrowly (ob1_mcp reads wiki + trellis for reflection synthesis; lovebug is RW everywhere). Isolation is enforced by Postgres at the role level, not by trusting MCP code to behave — matching the same "separate the *who*, not the *where*" pattern used for host users and Authentik. + +The embedding model is **local Ollama `nomic-embed-text` (768-d)** — never a hosted provider for memory. Switching it is a deliberate full re-embed, which is why `embedding_model` is part of the uniqueness key. + +### OB1's own two layers + +OB1 now mirrors the wiki's Sources/Wiki split internally: `ob1.thoughts` is the raw capture layer; `ob1.reflections` is a compiled layer written by a **daily reflection job** (`claude-sonnet-4-6` via the Anthropic Messages API direct) that synthesizes generalizable lessons with citations back to source rows. Same discipline as the wiki — mixing raw and synthesized content in one table pollutes semantic neighbors. Ambient capture is the [[Sources/Homelab/ob1-jsonl-watcher|JSONL watcher]]: it tails Claude Code / Cowork session files and posts turn-pairs to OB1's `capture_thought`. "Save everything, filter later." + +### Trellis: the third leg + +The original two-system framing (wiki + OB1) is now three: [[Sources/Homelab/trellis-mcp|Trellis]] is resumable *working state* — named threads with checkpoints that survive conversation boundaries. The clean separation the stack settled on: **CLAUDE.md = instructions (how to behave), Trellis = record (where we are in the doing), wiki + OB1 = lessons and outcomes (what we learned/built).** None should swallow another's job. Trellis lives in `petalbrain.trellis`; its tool surface is being refined ([[Sources/Homelab/trellis-mcp-tool-surface-followups|checkpoint_thread rename + id addressing]]). ### The Three Memory Layers @@ -38,11 +74,11 @@ Layer 1: Structured Wiki (wiki-vault) - Git-versioned - Written by: MCP server, email pipeline, Travis/Jenny -Layer 2: Semantic Memory (OB1) - - Supabase + pgvector - - Embeddings of vault content + session context - - Queryable by AI agents via semantic search - - Written by: n8n parallel pipeline (mirrors vault writes) +Layer 2: Semantic Memory (OB1) [see "Consolidated Substrate" above for current state] + - self-hosted pgvector:pg17 (petalbrain), NOT Supabase + - Embeddings of vault content + session context (Ollama nomic-embed-text, 768-d) + - Queryable by AI agents via semantic search over public.embeddings + - Written by: JSONL session watcher → capture_thought (NOT n8n parallel write) - Access: Tailscale-only Layer 3: Agent Context (Dispatch / Lovebug) @@ -101,10 +137,19 @@ The homelab MCP server (`mcp.herbylab.dev`, deployed 2026-05-06) is the critical **wiki-vault-migration** documents the vault consolidation from pbs-projects + homelab-projects into this unified structure, establishing the wiki-vault as the canonical structured knowledge store. +**postgres-consolidation-ob1-reflection-layer** (project plan) is the source for the single-database / three-schema / four-service-user model, the shared `public.embeddings` design, and OB1's two-layer captures+reflections split. **2026-05-15-postgres-consolidation-phase0-recon** is the primary evidence that OB1 is pgvector:pg17 + Ollama `nomic-embed-text` (768-d), not Supabase — it inventories the live instance directly. **phase2a-result** confirms the schemas, roles, and `public.embeddings` table landed with verified permission isolation. **postgres-consolidation-followups** supplies the reflection-model choice (`claude-sonnet-4-6` direct), chunking parameters (512/64), and the enricher loop bug. + +**ob1-jsonl-watcher** (project plan) is the source for the ambient-capture-via-JSONL design that replaced the n8n parallel-write idea. **trellis-mcp** establishes the third memory leg and the instructions/record/lessons three-way separation. **herbylab-project-management** documents the Postgres-backed artifact model and the NOTIFY→n8n→Trello projection. **vault-semantic-search** specifies the (still-unbuilt) `vault:search` tool over the wiki schema. + ## Open questions -- OB1 Phases 0–6 deployment timeline: what is the current blocker? The addendum was written but the deployment plan doesn't indicate Phase 0 is complete. -- When the n8n parallel write pipeline goes live, will it backfill existing vault content (all 62+ notes) into OB1, or only new writes going forward? -- The Dispatch pre-compact hook (Phase 7) depends on Dispatch exposing a lifecycle event. Is this a Dispatch feature that already exists, or does it require Dispatch development work? -- How will the MCP server's OB1 recall tool handle stale embeddings when vault content is updated by the compile loop? Will there be a re-embed step in the compile procedure? -- The wiki + OB1 parallel write model creates two sources of truth for the same content. Reconciliation strategy (wiki as authoritative; OB1 as derived index) should be documented once the pipeline goes live. +*Several earlier open questions were resolved by the May-2026 consolidation: OB1's substrate is confirmed (pgvector:pg17 / `petalbrain`, not Supabase); ambient capture is the JSONL watcher, not n8n parallel-write (so there is no wiki↔OB1 dual-source-of-truth problem — vault is content, OB1 indexes session content); and the shared `public.embeddings` table is live.* + +Still open: + +- **Reflector quality** — the daily reflection job's synthesis prompt is unproven; first non-dry-run output needs review before the 3am timer is trusted (postgres-consolidation-followups Phase 1). +- **`ob1-enricher` PENDING_QUERY loop** — must be fixed before the enricher is restarted, or the metadata backlog grows. +- **Re-embed on compile** — when the wiki compile loop edits a Wiki page, its `public.embeddings` row goes stale. There is still no re-embed step in the compile procedure. *(Agent inference, not stated in any source — flagged.)* +- **`vault:search` not yet built** — the semantic-search MCP tool over the wiki schema is designed ([[Sources/Homelab/vault-semantic-search|vault-semantic-search]]) but unbuilt; until it ships, the wiki half of the substrate isn't actually queryable by agents. +- **claude/competent-knuth-de9483 branch** — artifact tooling that landed via a separate path; merge/abandon/cherry-pick still undecided. +- **Dispatch pre-compact hook (Phase 7)** — still depends on a Dispatch lifecycle event; the JSONL watcher covers Claude Code/Cowork capture in the meantime. diff --git a/Wiki/Dev/pbs-hub.md b/Wiki/Dev/pbs-hub.md index deff352..401d364 100644 --- a/Wiki/Dev/pbs-hub.md +++ b/Wiki/Dev/pbs-hub.md @@ -5,7 +5,7 @@ tags: - pbs - python created: 2026-05-08 -updated: 2026-05-08 +updated: 2026-05-24 --- # pbs-hub @@ -24,6 +24,11 @@ In active development (Phase 5 of Content Hub). Architecture decisions locked: Phase 5 merges `pbs-api` + Content Hub codebases. Prior phases (1–4) built the Instagram automation database schema ("Great Lockout of 2026" workaround), reel detection system, and content tracking. +As of May 2026 the tool (codebase `pbs-video-manager`, served at `/pbscontenthub`) is being extended **upstream into pre-production** and given external write paths: +- **Pre-production extension** ([[Sources/Dev/pbs-hub-scene-management|scene-management]] / [[Sources/Dev/pbs-hub-pre-production-extension|pre-production-extension]]) — Outline/Treatment/Scenes authoring, a `scenes`/`scene_items` model (Path B: separate from `checklist_items`), a persistent card-rail + canvas UI, and a tablet **Present Mode** with color-coded scene completion to stop forgetting shots mid-shoot. All new tables target the `pbs_automation` DB. +- **Data import + MCP** ([[Sources/Dev/pbs-hub-data-import-and-mcp|data-import-and-mcp]]) — a JSON paste-import affordance plus a standalone **`pbs-hub-mcp`** server (its own repo/container) exposing additive-upsert + read/search tools to LLM clients. First homelab MCP that fronts a separate REST API; hand-mirrored Pydantic models with drift-detection CI. +- **Vector search** ([[Sources/Dev/pbs-hub-vector-search|vector-search]]) — semantic `search_projects` via LanceDB + OpenAI `text-embedding-3-small`, n8n-driven async embedding. Deferred ~18 months until the project library is large enough for title search to hurt. + ## Where it lives - PBS Linode Docker stack (`pbs-hub` service) @@ -41,6 +46,10 @@ Phase 5 merges `pbs-api` + Content Hub codebases. Prior phases (1–4) built the - [[Sources/Dev/instagram-automation-session-notes|instagram-automation-session-notes]] — schema created - [[Sources/Dev/instagram-reel-sync-phase4|instagram-reel-sync-phase4]] — reel detection system - [[Sources/Dev/pbs-api-db-connection-fix|pbs-api-db-connection-fix]] — pool_pre_ping connection fix +- [[Sources/Dev/pbs-hub-scene-management|pbs-hub-scene-management]] — pre-production scenes + Present Mode +- [[Sources/Dev/pbs-hub-pre-production-extension|pbs-hub-pre-production-extension]] — rail/canvas UI scope +- [[Sources/Dev/pbs-hub-data-import-and-mcp|pbs-hub-data-import-and-mcp]] — paste-import + pbs-hub-mcp +- [[Sources/Dev/pbs-hub-vector-search|pbs-hub-vector-search]] — semantic search (deferred) - [[Wiki/Reference/plant-based-southerner|plant-based-southerner]] — parent entity ## History @@ -49,3 +58,5 @@ Phase 5 merges `pbs-api` + Content Hub codebases. Prior phases (1–4) built the - **2026** — Three-tier reel detection system designed (Phase 4) - **2026** — DB connection fix: `pool_pre_ping=True`, `pool_recycle=1800` - **2026** — Phase 5 architecture planned: merge pbs-api + Content Hub into Blueprint Flask app +- **2026-05-17** — Pre-production extension planned: scenes/scene-items model, rail+canvas UI, tablet Present Mode (targets `pbs_automation`) +- **2026-05-19** — `pbs-hub-mcp` + JSON paste-import planned; vector-search spun out to its own (deferred) plan diff --git a/Wiki/Homelab/proxmox.md b/Wiki/Homelab/proxmox.md index 4321a6c..8698119 100644 --- a/Wiki/Homelab/proxmox.md +++ b/Wiki/Homelab/proxmox.md @@ -5,7 +5,7 @@ tags: - proxmox - homelab created: 2026-05-08 -updated: 2026-05-08 +updated: 2026-05-24 --- # Proxmox @@ -20,10 +20,13 @@ Active. Key guests running on Proxmox: - **ustest1** — Self-hosted GitHub Actions runner for PBS CI/CD (`pbsii-cicd-pipeline`) - **DNS stack** — Knot DNS primary + secondary, Pi-hole (split-horizon `.herbylab.dev`) - **Authentik** — SSO/IdP for homelab services -- **OB1** — Supabase + pgvector memory system (deployment in progress) +- **`homelab-postgres`** — pgvector:pg17 container hosting the consolidated `petalbrain` DB (wiki/ob1/trellis schemas) — the substrate for OB1 and the MCP servers +- **Traefik VM 102** (`10.0.21.181`) — homelab ingress with the `*.herbylab.dev` wildcard +- **PBS LXC 110** — Proxmox Backup Server (Debian 12); daily backups to local datastore, weekly rsync to NAS +- **us-test-authy (VM 100)** — recovered to static `10.0.21.207`; runs Claude Desktop / Dispatch - **herbydev** — Primary dev workstation VM (promoted from us-test-authy) -VLAN scheme renumbered in May 2026: VID 11 (lab network), VID 21 (private network); OpenWrt switch chip aligned to bridge VID assignment. +VLAN scheme renumbered in May 2026: VID 11 (lab network), VID 21 (private network); OpenWrt switch chip aligned to bridge VID assignment. (Note: DaVinci Resolve runs on the separate `herbyeos` Endeavour tower, not on Proxmox.) ## Where it lives @@ -39,6 +42,8 @@ VLAN scheme renumbered in May 2026: VID 11 (lab network), VID 21 (private networ - [[Sources/Homelab/dns-knot-pihole-end-to-end|dns-knot-pihole-end-to-end]] — DNS stack on Proxmox - [[Sources/Homelab/2026-05-08-homelab-vlan-renumber-switch-chip-to-bridge-alignment|2026-05-08-homelab-vlan-renumber-switch-chip-to-bridge-alignment]] — VLAN renumber - [[Sources/Homelab/homelab-mcp-server|homelab-mcp-server]] — MCP server (Proxmox-hosted) +- [[Sources/Homelab/postgres-consolidation-ob1-reflection-layer|postgres-consolidation-ob1-reflection-layer]] — `petalbrain` on `homelab-postgres` +- [[Sources/Homelab/2026-05-22-pbs-nas-off-site-backup-via-rsync|2026-05-22-pbs-nas-off-site-backup-via-rsync]] — PBS LXC 110 backup pipeline - [[Wiki/Homelab/homelab-ingress-stack|homelab-ingress-stack]] — synthesis ## History @@ -48,3 +53,5 @@ VLAN scheme renumbered in May 2026: VID 11 (lab network), VID 21 (private networ - **2026** — herbydev promoted from us-test-authy (Proxmox VM rename/reconfigure) - **2026-05-06** — MCP server deployed on Proxmox behind Authentik + Cloudflare Tunnel - **2026-05-08** — VLAN scheme renumbered (VID 11 lab, VID 21 private); switch chip → bridge alignment fixed +- **2026-05-15** — `petalbrain` consolidation: wiki/ob1/trellis schemas on the `homelab-postgres` pgvector:pg17 container +- **2026-05-22** — PBS LXC 110 (Debian 12) off-site backup pipeline; PVE→PBS→NAS rsync; day-1 restore validated diff --git a/Wiki/Reference/gitea.md b/Wiki/Reference/gitea.md index 9010a5b..56f7695 100644 --- a/Wiki/Reference/gitea.md +++ b/Wiki/Reference/gitea.md @@ -6,7 +6,7 @@ tags: - git - docker created: 2026-05-08 -updated: 2026-05-08 +updated: 2026-05-24 --- # Gitea @@ -24,6 +24,8 @@ Active. Key roles: The GitHub deploy refactor project completed the migration to pbsdeploybot with Gitea SSH host aliases, replacing the previous ambiguous key setup. +A **relocation is planned** ([[Sources/Homelab/gitea-migration-to-pve|gitea-migration-to-pve]]): promote a new PVE LXC Gitea to canonical for homelab/`petal-power` repos, keep the PBS instance canonical for PBS business code, and cross-mirror each as the other's insurance copy. The reframe is that PBS-Gitea was already serving as paid offsite backup. Lovebug also pushes vault-mcp commits to Gitea over HTTPS using a `lovebug` PAT via an env-driven git credential helper (the semicolon-trap and token fixes are documented in the consolidation session notes). + ## Where it lives - PBS Linode Docker stack (`gitea` service) @@ -36,6 +38,7 @@ The GitHub deploy refactor project completed the migration to pbsdeploybot with - [[Sources/Dev/email-to-obsidian-automation|email-to-obsidian-automation]] — Gitea as vault write endpoint - [[Sources/Dev/email-obsidian-git-pipeline|email-obsidian-git-pipeline]] — email pipeline session notes - [[Sources/Dev/env-file-hardening|env-file-hardening]] — pbsdeploy credential separation +- [[Sources/Homelab/gitea-migration-to-pve|gitea-migration-to-pve]] — planned PVE LXC relocation + cross-mirror - [[Wiki/Reference/traefik|traefik]] — ingress layer - [[Wiki/Reference/n8n|n8n]] — automation that pushes to Gitea @@ -44,3 +47,5 @@ The GitHub deploy refactor project completed the migration to pbsdeploybot with - Ongoing — Self-hosted git service in PBS stack - **2026** — pbsdeploybot machine user created; SSH host alias pattern implemented - **2026** — Email-to-vault pipeline wired through Gitea as the push endpoint +- **2026-05** — vault-mcp push surface wired (lovebug PAT + HTTPS credential helper) +- **2026-05-23** — PVE LXC migration plan written: two-instance cross-mirror, `petal-power` canonical on the new instance diff --git a/Wiki/Reference/index.md b/Wiki/Reference/index.md index 77ac598..73522ba 100644 --- a/Wiki/Reference/index.md +++ b/Wiki/Reference/index.md @@ -3,7 +3,7 @@ type: topic-landing path: Wiki/Reference tags: - landing -updated: 2026-05-08 +updated: 2026-05-24 --- # Reference — Wiki @@ -21,7 +21,7 @@ Compiled entities, topic landings, and synthesis for the Reference domain. **AI & Agents** - [[Wiki/Reference/lovebug|lovebug]] — named AI coding agent; Dispatch instance; primary collaborator -- [[Wiki/Reference/ob1|ob1]] — AI memory system; Supabase + pgvector; Tailscale-only; Phases 0–6 +- [[Wiki/Reference/ob1|ob1]] — AI memory system; self-hosted pgvector:pg17 (`petalbrain`) + Ollama; Tailscale-only - [[Wiki/Reference/claude-code|claude-code]] — Anthropic CLI; primary AI tool across all projects - [[Wiki/Reference/dispatch|dispatch]] — AI agent orchestration framework; manages Lovebug lifecycle diff --git a/Wiki/Reference/ob1.md b/Wiki/Reference/ob1.md index ca990b9..d6ce4c6 100644 --- a/Wiki/Reference/ob1.md +++ b/Wiki/Reference/ob1.md @@ -7,45 +7,49 @@ tags: - pgvector - automation created: 2026-05-08 -updated: 2026-05-08 +updated: 2026-05-24 --- # OB1 ## What it is -OB1 is Travis's personal AI memory and retrieval system — a Supabase + pgvector stack that stores embeddings of project notes, session logs, and vault content to enable semantic recall across AI conversations. Named as a reference to the "Old Ben" Kenobi archetype of a wise, reliable memory keeper. OB1 sits alongside the wiki-vault in the broader second-brain architecture: the vault provides structured human-readable knowledge, OB1 provides fast semantic search for AI context injection. +OB1 (Open Brain) is Travis's personal AI memory and retrieval system — a self-hosted **pgvector/pgvector:pg17** Postgres store holding embeddings of session content, project notes, and vault content for semantic recall across AI conversations. It is the fast, voluminous, agent-facing half of the two-system memory model; the [[Wiki/Dev/ai-memory-architecture|wiki-vault]] is the slow, curated, human-facing half. OB1 is forked from the upstream community project `NateBJones-Projects/OB1`. + +> **Correction (2026-05-24 compile):** earlier versions of this page described OB1 as a "Supabase + pgvector" stack. The postgres-consolidation recon proved that's wrong — OB1 has always been a self-hosted pgvector container (`homelab-postgres`, DB now `petalbrain`), with embeddings from local **Ollama `nomic-embed-text`** (768-d), never Supabase. ## Current state -In active deployment across Phases 0–6. Key architectural decisions (from the addendum): -- **Tailscale-only access** — Phase 4 deferred public endpoint; OB1 only reachable over Tailscale mesh -- **Parallel write pipeline** — n8n writes new vault notes to both wiki-vault and OB1 simultaneously -- **Phase 7 (planned)** — Dispatch pre-compact hook wires Lovebug's context windows into OB1 for automatic memory ingestion before context is dropped - -The AI memory architecture research session explored harness-vs-model layering, Poolside and Pi.dev approaches, and the split between episodic (session notes) and semantic (wiki entity) memory modes. +Operational and consolidated into the `petalbrain` Postgres instance. Key facts as of the May 2026 consolidation: +- **Storage** — `ob1.thoughts` (raw captures) + `ob1.reflections` (synthesized lessons); embeddings live in the shared `public.embeddings` table (HNSW, vector_cosine_ops, m=16/ef_construction=64), 512-token chunks / 64-token overlap +- **Two-layer model** — raw captures + a daily reflection job (`claude-sonnet-4-6` via Anthropic API direct) that synthesizes generalizable lessons, mirroring the wiki's Sources/Wiki split +- **Ambient capture** — the [[Sources/Homelab/ob1-jsonl-watcher|ob1-jsonl-watcher]] tails Claude Code / Cowork JSONL session files and posts turn-pairs to OB1's `capture_thought` (replaces the earlier "n8n parallel write" idea) +- **Embedding model** — Ollama `nomic-embed-text` (768-d), local; switching requires a full re-embed +- **Tailscale-only access** — no public endpoint +- **`ob1-watcher` / `ob1-enricher`** daemons; enricher has a known PENDING_QUERY loop bug pending a fix before restart ## Where it lives -- **Supabase instance** — hosted in homelab (Proxmox), contains pgvector extension -- **Ingest pipeline** — n8n on PBS Docker stack -- **Query interface** — planned: homelab MCP server exposes OB1 recall as a tool -- **Access** — Tailscale-only (no public endpoint) +- **Postgres** — `petalbrain` DB on `homelab-postgres` (pgvector:pg17), Proxmox; schema `ob1`, owned by the `ob1_mcp` service role +- **Embeddings** — Ollama (`nomic-embed-text`) on the homelab +- **Capture** — `ob1-jsonl-watcher` daemon on herbydev reading `~/.claude` session logs +- **Access** — Tailscale-only ## Related projects - [[Sources/Homelab/ob1-main-deployment|ob1-main-deployment]] — full deployment plan (Phases 0–6) -- [[Sources/Homelab/ob1-deployment|ob1-deployment]] — addendum (Tailscale, parallel write, Phase 7) +- [[Sources/Homelab/ob1-deployment|ob1-deployment]] — addendum (Tailscale, Phase 7 Dispatch hook) +- [[Sources/Homelab/ob1-jsonl-watcher|ob1-jsonl-watcher]] — ambient session capture +- [[Sources/Homelab/postgres-consolidation-ob1-reflection-layer|postgres-consolidation-ob1-reflection-layer]] — consolidation + reflection layer +- [[Sources/Homelab/postgres-consolidation-followups|postgres-consolidation-followups]] — reflector activation + cleanup - [[Sources/Dev/ai-memory-architecture-research|ai-memory-architecture-research]] — architecture research session -- [[Sources/Dev/second-brain|second-brain]] — wiki-vault + OB1 second-brain pattern -- [[Sources/Homelab/homelab-mcp-server|homelab-mcp-server]] — MCP server will expose OB1 recall -- [[Wiki/Reference/n8n|n8n]] — ingest pipeline - [[Wiki/Reference/tailscale|tailscale]] — access layer - [[Wiki/Dev/ai-memory-architecture|ai-memory-architecture]] — synthesis page ## History -- **2026** — Architecture research session: pgvector, harness vs model layers, Poolside/Pi.dev reviewed -- **2026** — Full deployment plan written (Phases 0–6, Supabase + pgvector) -- **2026** — Addendum locked: Tailscale-only, parallel n8n write, Phase 7 Dispatch hook -- **Planned** — MCP server integration to expose OB1 recall as homelab tool +- **2026** — Architecture research: pgvector, harness vs model layers, Poolside/Pi.dev reviewed +- **2026** — Deployment (Phases 0–6); pgvector container + Ollama embeddings; Tailscale-only +- **2026-05-14** — Embedding dim migrated 1536→768 for `nomic-embed-text`; `ob1-jsonl-watcher` plan written +- **2026-05-15** — Consolidated into `petalbrain`; `ob1.thoughts` + shared `public.embeddings`; reflection layer designed +- **2026-05-18** — Dispatch subprocess-leak incident reinforced the case for harness-owned subprocess lifecycle diff --git a/Wiki/Reference/traefik.md b/Wiki/Reference/traefik.md index 1c70ac4..d5e58ed 100644 --- a/Wiki/Reference/traefik.md +++ b/Wiki/Reference/traefik.md @@ -6,7 +6,7 @@ tags: - docker - homelab created: 2026-05-08 -updated: 2026-05-08 +updated: 2026-05-24 --- # Traefik @@ -24,7 +24,7 @@ Active at version 3.6.11 (upgraded March 2026 during the server stability harden - **Cloudflare Tunnel** — Traefik sits behind Cloudflare Tunnel for MCP and some homelab services - **Knot DNS** — primary and secondary DNS integrated with Traefik TLS pipeline -The homelab also runs Traefik (or uses the same pattern) for `.herbylab.dev` services behind Authentik. +The homelab runs its own Traefik **3.7 on VM 102 (`10.0.21.181`)** with Docker + file + ACME providers, separate from the PBS instance. It holds the LE prod wildcard `*.herbylab.dev` (Cloudflare DNS-01) and fronts a growing set of services — `whoami`, the Traefik dashboard (TLS + basicauth), `pihole`, and `openwrt` LuCI — each reachable behind a `cloudflared` tunnel (`traefik-vm`). ## Where it lives @@ -39,6 +39,8 @@ The homelab also runs Traefik (or uses the same pattern) for `.herbylab.dev` ser - [[Sources/Homelab/wp-cron-supercronic-deploy|wp-cron-supercronic-deploy]] — block-wpcron middleware - [[Sources/Homelab/ufw-docker-outage-fix|ufw-docker-outage-fix]] — UFW + Docker/Traefik outbound rules fix - [[Sources/Homelab/homelab-mcp-server|homelab-mcp-server]] — MCP server behind Traefik +- [[Sources/Homelab/2026-05-11-traefik-phase1-phase2-wildcard-canary|2026-05-11-traefik-phase1-phase2-wildcard-canary]] — homelab VM 102: wildcard cert + canary +- [[Sources/Homelab/2026-05-21-traefik-routes-for-pihole-openwrt-us-test-authy-static-ip-cutove|2026-05-21-traefik-routes-pihole-openwrt]] — pihole/openwrt routes - [[Wiki/Reference/cloudflare|cloudflare]] — tunnel + DNS - [[Wiki/Reference/authentik|authentik]] — auth layer - [[Wiki/Homelab/homelab-ingress-stack|homelab-ingress-stack]] — synthesis @@ -50,3 +52,5 @@ The homelab also runs Traefik (or uses the same pattern) for `.herbylab.dev` ser - **March 2026** — `block-wpcron` middleware added when Supercronic replaced WP-Cron - **2026** — UFW `after.rules` fixed to restore Docker/Traefik outbound routing after UFW misconfiguration caused outage - **2026-05-06** — Traefik used as ingress for MCP server at `mcp.herbylab.dev` +- **2026-05-11** — Homelab Traefik 3.7 on VM 102: Phase 1.6 wildcard cert + Phase 2 canary; dashboard locked behind TLS+basicauth; `traefik-vm` cloudflared tunnel +- **2026-05-21** — `pihole` + `openwrt` routed on `*.herbylab.dev`; `addPrefix` redirect-loop and self-signed-backend (`insecureSkipVerify`) lessons captured diff --git a/Wiki/index.md b/Wiki/index.md index 78e630c..33d42e9 100644 --- a/Wiki/index.md +++ b/Wiki/index.md @@ -3,7 +3,7 @@ type: topic-landing path: Wiki tags: - landing -updated: 2026-05-08 +updated: 2026-05-24 --- # Wiki @@ -21,7 +21,7 @@ Generated and maintained by the wiki-maintenance compile skill. ## Key Entities - [[Wiki/Reference/lovebug|lovebug]] — named AI coding agent (Dispatch) -- [[Wiki/Reference/ob1|ob1]] — AI memory system (Supabase + pgvector) +- [[Wiki/Reference/ob1|ob1]] — AI memory system (self-hosted pgvector:pg17 / `petalbrain`) - [[Wiki/Reference/plant-based-southerner|plant-based-southerner]] — the PBS business - [[Wiki/Reference/jenny|jenny]] — PBS co-founder and content creator - [[Wiki/Reference/n8n|n8n]] — central automation hub diff --git a/log.md b/log.md index fb42f8e..1b0bb00 100644 --- a/log.md +++ b/log.md @@ -24,3 +24,25 @@ Append-only event log. Each entry records a compile, lint, or migration run. Sources/index, Wiki/Dev, Wiki/Venture, Wiki/Homelab, Wiki/Reference, Wiki/index, root index - Entity promotions: all 17 met 2+ project threshold; Docker and Dispatch added per user spec - No pages deleted or demoted + +## [2026-05-24 21:30] compile | 34 sources since first compile (11 Dev, 23 Homelab) + +- Pages touched: Sources/index, Sources/Dev/index, Sources/Homelab/index, Wiki/index, + Wiki/Reference/index, Wiki/Reference/ob1, Wiki/Reference/traefik, Wiki/Reference/gitea, + Wiki/Homelab/proxmox, Wiki/Dev/pbs-hub, Wiki/Dev/ai-memory-architecture (synthesis) +- Pages created: none (entity promotions proposed, not auto-created — see below) +- Entities promoted: none this run +- Entities PROPOSED for promotion (await user approval): trellis, vault-mcp, petalbrain, + pbs-hub-mcp, knot-dns, herbyeos +- Synthesis pages written: 0 new; ai-memory-architecture substantially updated (petalbrain + consolidation, public.embeddings, OB1 two-layer + reflection job, JSONL watcher, Trellis + third leg); 9→17 sources +- Contradictions flagged: OB1 described as "Supabase + pgvector" across ob1 entity, Wiki/index, + Wiki/Reference/index, ai-memory-architecture — corrected to self-hosted pgvector:pg17/petalbrain + + Ollama nomic-embed-text (per phase0-recon direct inventory) +- Tags proposed: none +- Notes: 8 source files are pipeline test artifacts (5x vault-push/create-artifact verification, + 2x vault-integration-test, 1x vault-semantic-search-integration-test) — flagged for cleanup, + NOT deleted. Lint items: type=session-note (singular) on 4 files should be session-notes; + MCP-written project-plans intentionally omit status/updated frontmatter (per herbylab-project- + management design) — not an error. Venture domain unchanged (no new sources).