second-brain

A personal knowledge system: video/article extraction pipeline + LLM-maintained wiki.

Architecture

Three-layer system:

Wiki — long-term structured memory. Markdown files in an Obsidian vault, maintained by the wiki compiler.
Context assembler — builds the prompt bundle for each extraction. Phase 1 is thin: domain template + focus field + transcript.
Extractor — stateless single-shot LLM call per source. Pure function: bundle in, structured extraction out.

Pipeline stages

pending → pulled → transcribed → analyzed → accepted → published

Project layout

src/second_brain/
├── adapters/      # Pull adapters (youtube.py, article.py)
├── context/       # Context assembler
├── extractor/     # LLM extraction engine + Pydantic schema
├── llm/           # claude_cli.py — thin subprocess wrapper around `claude -p`
├── compiler/      # Wiki compiler (reads vault, writes pages, git commits)
├── scheduler/     # Rate-limit smoother (call-count for CLI, token-budget for API)
└── web/           # FastAPI + Jinja2 + HTMX UI

Setup

cp config/settings.toml.example config/settings.toml
# Edit settings.toml with your paths
uv run second-brain serve

CLI commands

second-brain add <url> — queue a source URL (video or article)
second-brain process — run the pipeline on pending items
second-brain serve — launch the web UI (default port 8000)
second-brain compile — run the wiki compiler
second-brain schedule — start the overnight scheduler

Extractor backends

[extractor].backend in config/settings.toml picks the LLM transport.

cli (default) — src/second_brain/llm/claude_cli.py shells out to claude -p --output-format json --model <model> --json-schema <schema>. Runs under the Max OAuth subscription (no per-token billing). Hermetic isolation: spawned inside tempfile.TemporaryDirectory(prefix="sb-claude-cli-"), with ANTHROPIC_API_KEY / ANTHROPIC_AUTH_TOKEN stripped from env so the CLI uses OAuth instead of an API key, and with --no-session-persistence --disable-slash-commands --tools "" --setting-sources "" so host CLAUDE.md / hooks / agents don't leak in.
api — uses the anthropic SDK. Lazy-imported only when this backend is selected. The SDK is an optional dependency (uv sync --extra api).

Schema-validated output gotcha: when --json-schema is used, the parsed object lands at payload["structured_output"], not payload["result"]. The wrapper checks both. The engine prefers the structured payload via LLMExtraction.model_validate(structured) and falls back to parsing the text field.

Environment variables

ANTHROPIC_API_KEY — only required when extractor.backend = "api". The default cli backend uses your Max OAuth and does not need it.
SECOND_BRAIN_CONFIG — optional override for the settings file path.

Domains

development — software, programming, systems
content — content creation strategy, video production
business — entrepreneurship, marketing, operations
homelab — self-hosted infrastructure, networking, DevOps

Key files

config/settings.toml — vault path, DB path, scheduler window, [extractor] backend config
prompts/<domain>.md — per-domain extraction prompt templates
src/second_brain/extractor/schema.py — LLMExtraction (LLM-only fields, drives the --json-schema contract) + ExtractionResult (LLMExtraction + SourceMeta filled from the DB)
src/second_brain/llm/claude_cli.py — CLI subprocess wrapper
src/second_brain/models.py — SQLAlchemy models + utcnow() helper (replaces deprecated datetime.utcnow)
postgres-migration-planning.md — open questions blocking the SQLite → Postgres+pgvector migration

Conventions / gotchas

Naive UTC timestamps everywhere. Use from second_brain.models import utcnow rather than datetime.utcnow() (deprecated in 3.12) or datetime.now() (local time).
Re-query inside the session. SQLAlchemy ORM attribute access after the session closes raises DetachedInstanceError. The process command and the web accept/reject endpoints both collect IDs first, then re-query inside the per-iteration / per-request session block, and pre-compute any dict views needed for templates inside the with db.session() block.
Starlette TemplateResponse signature. Starlette ≥ 0.36 requires templates.TemplateResponse(request, name, context). The old (name, context) form raises TypeError: unhashable type: 'dict'. Do not put "request": request into the context dict — it's auto-injected now.
Vault git repo auto-init. The wiki compiler calls _ensure_git_repo() on every run, which git inits the vault and creates an empty seed commit if no .git is present. _git_commit() also checks git diff --cached --quiet before committing to avoid empty commits.
Scheduler throttling shape. The CLI backend is throttled by max_calls_per_hour (Max subscription limits are call-shaped, not token-shaped). The API backend uses max_tokens_per_hour. Both share min_gap_seconds. The scheduler targets sources in TRANSCRIBED state — earlier code mistakenly filtered on ANALYZED, making it a no-op.

In-flight work

Postgres + pgvector migration. SQLite is the current backing store. Travis wants to move to his existing Postgres + pgvector instance and use the shared /projects/shared/python/embedding_chunking module for vector indexing. Planning questions are enumerated in postgres-migration-planning.md at the project root; no migration work has started yet.
Vault location. Recommended /opt/projects/second-brain-vault/ as the production location (peer to the project dir). Not yet confirmed.

Validation

Per Travis's global CLAUDE.md, run zero-check after any code change, and a-review after non-trivial diffs (> 50 lines). a-review script: python3 ~/.claude/skills/a-review/run_review.py, output goes to reviews/a-review-<date>.md.

6.0 KiB Raw Blame History