second-brain/CLAUDE.md

second-brain

A personal knowledge system: video/article extraction pipeline + LLM-maintained wiki.

Architecture

Three-layer system:

1. Wiki — long-term structured memory. Markdown files in an Obsidian vault, maintained by the wiki compiler.
2. Context assembler — builds the prompt bundle for each extraction. Phase 1 is thin: domain template + focus field + transcript.
3. Extractor — stateless single-shot LLM call per source. Pure function: bundle in, structured extraction out.

Pipeline stages

pending → pulled → transcribed → analyzed → accepted → published

Project layout

src/second_brain/
├── adapters/      # Pull adapters (youtube.py, article.py)
├── context/       # Context assembler
├── embeddings/    # Chunk + nomic-embed-text + upsert into public.embeddings
├── extractor/     # LLM extraction engine + Pydantic schema
├── llm/           # claude_cli.py — thin subprocess wrapper around `claude -p`
├── compiler/      # Wiki compiler (reads vault, writes pages, git commits)
├── scheduler/     # Rate-limit smoother (call-count for CLI, token-budget for API)
└── web/           # FastAPI + Jinja2 + HTMX UI

alembic/           # Schema migrations for the second_brain Postgres schema

Setup

cp config/settings.toml.example config/settings.toml
# Edit settings.toml — at minimum, point the paths and set SECOND_BRAIN_DATABASE_URL.

# Connection string format. Containerised (default):
#   postgresql+psycopg://lovebug:<password>@homelab-postgres:5432/petalbrain
# Host-side dev runs:
#   postgresql+psycopg://lovebug:<password>@127.0.0.1:5433/petalbrain
export SECOND_BRAIN_DATABASE_URL=...

# One-time, before first migration: bootstrap the schema as postgres
# superuser. lovebug doesn't have CREATE on petalbrain, so this can't be
# done from the app side — same pattern as wiki/trellis/ob1.
#   CREATE SCHEMA second_brain AUTHORIZATION lovebug;
#   GRANT ALL PRIVILEGES ON SCHEMA second_brain TO lovebug;

uv sync
uv run alembic upgrade head
uv run second-brain serve

CLI commands

• second-brain add <url> — queue a source URL (video or article)
• second-brain process — run the pipeline on pending items
• second-brain serve — launch the web UI (default port 8000)
• second-brain compile — run the wiki compiler
• second-brain schedule — start the overnight scheduler

Extractor backends

[extractor].backend in config/settings.toml picks the LLM transport.

• cli (default) — src/second_brain/llm/claude_cli.py shells out to claude -p --output-format json --model <model> --json-schema <schema>. Runs under the Max OAuth subscription (no per-token billing). Hermetic isolation: spawned inside tempfile.TemporaryDirectory(prefix="sb-claude-cli-"), with ANTHROPIC_API_KEY / ANTHROPIC_AUTH_TOKEN stripped from env so the CLI uses OAuth instead of an API key, and with --no-session-persistence --disable-slash-commands --tools "" --setting-sources "" so host CLAUDE.md / hooks / agents don't leak in.
• api — uses the anthropic SDK. Lazy-imported only when this backend is selected. The SDK is an optional dependency (uv sync --extra api).

Schema-validated output gotcha: when --json-schema is used, the parsed object lands at payload["structured_output"], not payload["result"]. The wrapper checks both. The engine prefers the structured payload via LLMExtraction.model_validate(structured) and falls back to parsing the text field.

Environment variables

• ANTHROPIC_API_KEY — only required when extractor.backend = "api". The default cli backend uses your Max OAuth and does not need it.
• SECOND_BRAIN_CONFIG — optional override for the settings file path.
• SECOND_BRAIN_DATABASE_URL — Postgres connection string. Falls back to HERBYLAB_DATABASE_URL if unset. Required at runtime; both second_brain.database and second_brain.embeddings read it. Use the SQLAlchemy postgresql+psycopg:// form; the embeddings module strips the +psycopg prefix internally for raw psycopg.
• OLLAMA_URL — overrides [embeddings].ollama_url. Default http://ollama:11434 (homelab docker network); for host-side runs use http://127.0.0.1:11434.
• EMBEDDING_MODEL — overrides [embeddings].model. Default nomic-embed-text (768-d).
• SECOND_BRAIN_DB_POOL_MIN / SECOND_BRAIN_DB_POOL_MAX — SQLAlchemy pool sizing knobs (defaults 1 / 9 overflow).

Domains

• development — software, programming, systems
• content — content creation strategy, video production
• business — entrepreneurship, marketing, operations
• homelab — self-hosted infrastructure, networking, DevOps

Key files

• config/settings.toml — vault path, [database], [extractor], [embeddings], [scheduler]
• prompts/<domain>.md — per-domain extraction prompt templates
• src/second_brain/extractor/schema.py — LLMExtraction (LLM-only fields, drives the --json-schema contract) + ExtractionResult (LLMExtraction + SourceMeta filled from the DB)
• src/second_brain/llm/claude_cli.py — CLI subprocess wrapper
• src/second_brain/models.py — SQLAlchemy models (all tables live in the second_brain schema) + utcnow() helper
• src/second_brain/database.py — Postgres engine + URL resolution
• src/second_brain/embeddings/__init__.py — chunk → Ollama → upsert into public.embeddings; best-effort with graceful degradation
• alembic/ + alembic.ini — schema migrations for the second_brain schema (Alembic does not manage public.embeddings)
• postgres-migration-planning.md — original planning doc; superseded by this file once the migration shipped

Conventions / gotchas

• Naive UTC timestamps everywhere. Use from second_brain.models import utcnow rather than datetime.utcnow() (deprecated in 3.12) or datetime.now() (local time). Postgres columns are timestamp without time zone — do not switch to timestamptz.
• PK style. Int autoincrement (SERIAL) — locked for compatibility with the existing schema shape. Do not switch to BIGINT GENERATED ALWAYS AS IDENTITY or UUID.
• Re-query inside the session. SQLAlchemy ORM attribute access after the session closes raises DetachedInstanceError. The process command and the web accept/reject endpoints both collect IDs first, then re-query inside the per-iteration / per-request session block, and pre-compute any dict views needed for templates inside the with db.session() block.
• Starlette TemplateResponse signature. Starlette ≥ 0.36 requires templates.TemplateResponse(request, name, context). The old (name, context) form raises TypeError: unhashable type: 'dict'. Do not put "request": request into the context dict — it's auto-injected now.
• Vault git repo auto-init. The wiki compiler calls _ensure_git_repo() on every run, which git inits the vault and creates an empty seed commit if no .git is present. _git_commit() also checks git diff --cached --quiet before committing to avoid empty commits.
• Scheduler throttling shape. The CLI backend is throttled by max_calls_per_hour (Max subscription limits are call-shaped, not token-shaped). The API backend uses max_tokens_per_hour. Both share min_gap_seconds. The scheduler targets sources in TRANSCRIBED state — earlier code mistakenly filtered on ANALYZED, making it a no-op.
• Alembic + lovebug privilege gap. lovebug owns the second_brain schema but does not have CREATE on the petalbrain database. alembic/env.py therefore guards CREATE SCHEMA behind a pg_namespace lookup, and the v1 migration intentionally drops the IF NOT EXISTS form (a bare CREATE SCHEMA IF NOT EXISTS raises permission denied even when the schema is already present). The schema is bootstrapped once, out-of-band, by postgres superuser.
• public.embeddings is shared, not ours. Alembic only manages the second_brain schema. The vector table itself is owned by postgres and shared across vault-mcp / ob1 / second-brain — we just INSERT into it. Rows are keyed by (source_schema='second_brain', source_table='extractions', source_id, embedding_model='nomic-embed-text').
• Embedding is best-effort. The pipeline writes the relational extractions row first and sess.flush()es; only then calls embed_extraction(). Any failure (no DB URL, no Ollama, embedding 500) logs at WARNING and returns None — the relational row and file vault stay canonical.
• File-based wiki, not petalbrain.wiki. The wiki compiler still writes markdown into the configured vault_path and commits via git. We do not dual-write into petalbrain.wiki — that's vault-mcp's territory and intentionally out of scope.

In-flight work

• Vault location. Recommended /opt/projects/second-brain-vault/ as the production location (peer to the project dir). Not yet confirmed.

Validation

Per Travis's global CLAUDE.md, run zero-check after any code change, and a-review after non-trivial diffs (> 50 lines). a-review script: python3 ~/.claude/skills/a-review/run_review.py, output goes to reviews/a-review-<date>.md.