petal-power/second-brain
3
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
373e137993
second-brain/CLAUDE.md
Travis Herbranson 6c1445c8ed docs: update CLAUDE.md + migration plan to reflect shipped Postgres design 
CLAUDE.md updates:
- new src/second_brain/embeddings/ entry in the project layout
- setup section now lists the postgres-superuser bootstrap (CREATE SCHEMA
  AUTHORIZATION lovebug) and the alembic step
- env var table now covers SECOND_BRAIN_DATABASE_URL, OLLAMA_URL,
  EMBEDDING_MODEL, and the pool sizing knobs
- conventions section gained five new gotchas (the lovebug CREATE gap,
  public.embeddings ownership, best-effort embedding, etc.)

postgres-migration-planning.md flipped from "planning questions" to a
"what shipped" runbook — locked decisions table, operator setup steps,
known gotchas, and the next-iteration backlog.
2026-05-24 22:57:31 -04:00

123 lines
9.0 KiB
Markdown
Raw Blame History

# second-brain
A personal knowledge system: video/article extraction pipeline + LLM-maintained wiki.
## Architecture
Three-layer system:
1. **Wiki** — long-term structured memory. Markdown files in an Obsidian vault, maintained by the wiki compiler.
2. **Context assembler** — builds the prompt bundle for each extraction. Phase 1 is thin: domain template + focus field + transcript.
3. **Extractor** — stateless single-shot LLM call per source. Pure function: bundle in, structured extraction out.
## Pipeline stages
```
pending → pulled → transcribed → analyzed → accepted → published
```
## Project layout
```
src/second_brain/
├── adapters/ # Pull adapters (youtube.py, article.py)
├── context/ # Context assembler
├── embeddings/ # Chunk + nomic-embed-text + upsert into public.embeddings
├── extractor/ # LLM extraction engine + Pydantic schema
├── llm/ # claude_cli.py — thin subprocess wrapper around `claude -p`
├── compiler/ # Wiki compiler (reads vault, writes pages, git commits)
├── scheduler/ # Rate-limit smoother (call-count for CLI, token-budget for API)
└── web/ # FastAPI + Jinja2 + HTMX UI
alembic/ # Schema migrations for the second_brain Postgres schema
```
## Setup
```bash
cp config/settings.toml.example config/settings.toml
# Edit settings.toml — at minimum, point the paths and set SECOND_BRAIN_DATABASE_URL.
# Connection string format. Containerised (default):
# postgresql+psycopg://lovebug:<password>@homelab-postgres:5432/petalbrain
# Host-side dev runs:
# postgresql+psycopg://lovebug:<password>@127.0.0.1:5433/petalbrain
export SECOND_BRAIN_DATABASE_URL=...
# One-time, before first migration: bootstrap the schema as postgres
# superuser. lovebug doesn't have CREATE on petalbrain, so this can't be
# done from the app side — same pattern as wiki/trellis/ob1.
# CREATE SCHEMA second_brain AUTHORIZATION lovebug;
# GRANT ALL PRIVILEGES ON SCHEMA second_brain TO lovebug;
uv sync
uv run alembic upgrade head
uv run second-brain serve
```
## CLI commands
- `second-brain add <url>` — queue a source URL (video or article)
- `second-brain process` — run the pipeline on pending items
- `second-brain serve` — launch the web UI (default port 8000)
- `second-brain compile` — run the wiki compiler
- `second-brain schedule` — start the overnight scheduler
## Extractor backends
`[extractor].backend` in `config/settings.toml` picks the LLM transport.
- **`cli`** (default) — `src/second_brain/llm/claude_cli.py` shells out to `claude -p --output-format json --model <model> --json-schema <schema>`. Runs under the Max OAuth subscription (no per-token billing). Hermetic isolation: spawned inside `tempfile.TemporaryDirectory(prefix="sb-claude-cli-")`, with `ANTHROPIC_API_KEY` / `ANTHROPIC_AUTH_TOKEN` stripped from env so the CLI uses OAuth instead of an API key, and with `--no-session-persistence --disable-slash-commands --tools "" --setting-sources ""` so host CLAUDE.md / hooks / agents don't leak in.
- **`api`** — uses the `anthropic` SDK. Lazy-imported only when this backend is selected. The SDK is an optional dependency (`uv sync --extra api`).
**Schema-validated output gotcha:** when `--json-schema` is used, the parsed object lands at `payload["structured_output"]`, not `payload["result"]`. The wrapper checks both. The engine prefers the structured payload via `LLMExtraction.model_validate(structured)` and falls back to parsing the text field.
## Environment variables
- `ANTHROPIC_API_KEY` — only required when `extractor.backend = "api"`. The default `cli` backend uses your Max OAuth and does not need it.
- `SECOND_BRAIN_CONFIG` — optional override for the settings file path.
- `SECOND_BRAIN_DATABASE_URL` — Postgres connection string. Falls back to `HERBYLAB_DATABASE_URL` if unset. Required at runtime; both `second_brain.database` and `second_brain.embeddings` read it. Use the SQLAlchemy `postgresql+psycopg://` form; the embeddings module strips the `+psycopg` prefix internally for raw psycopg.
- `OLLAMA_URL` — overrides `[embeddings].ollama_url`. Default `http://ollama:11434` (homelab docker network); for host-side runs use `http://127.0.0.1:11434`.
- `EMBEDDING_MODEL` — overrides `[embeddings].model`. Default `nomic-embed-text` (768-d).
- `SECOND_BRAIN_DB_POOL_MIN` / `SECOND_BRAIN_DB_POOL_MAX` — SQLAlchemy pool sizing knobs (defaults 1 / 9 overflow).
## Domains
- `development` — software, programming, systems
- `content` — content creation strategy, video production
- `business` — entrepreneurship, marketing, operations
- `homelab` — self-hosted infrastructure, networking, DevOps
## Key files
- `config/settings.toml` — vault path, `[database]`, `[extractor]`, `[embeddings]`, `[scheduler]`
- `prompts/<domain>.md` — per-domain extraction prompt templates
- `src/second_brain/extractor/schema.py` — `LLMExtraction` (LLM-only fields, drives the `--json-schema` contract) + `ExtractionResult` (LLMExtraction + SourceMeta filled from the DB)
- `src/second_brain/llm/claude_cli.py` — CLI subprocess wrapper
- `src/second_brain/models.py` — SQLAlchemy models (all tables live in the `second_brain` schema) + `utcnow()` helper
- `src/second_brain/database.py` — Postgres engine + URL resolution
- `src/second_brain/embeddings/__init__.py` — chunk → Ollama → upsert into `public.embeddings`; best-effort with graceful degradation
- `alembic/` + `alembic.ini` — schema migrations for the `second_brain` schema (Alembic does **not** manage `public.embeddings`)
- `postgres-migration-planning.md` — original planning doc; superseded by this file once the migration shipped
## Conventions / gotchas
- **Naive UTC timestamps everywhere.** Use `from second_brain.models import utcnow` rather than `datetime.utcnow()` (deprecated in 3.12) or `datetime.now()` (local time). Postgres columns are `timestamp without time zone` — do not switch to `timestamptz`.
- **PK style.** Int autoincrement (`SERIAL`) — locked for compatibility with the existing schema shape. Do not switch to `BIGINT GENERATED ALWAYS AS IDENTITY` or UUID.
- **Re-query inside the session.** SQLAlchemy ORM attribute access after the session closes raises `DetachedInstanceError`. The `process` command and the web accept/reject endpoints both collect IDs first, then re-query inside the per-iteration / per-request session block, and pre-compute any dict views needed for templates *inside* the `with db.session()` block.
- **Starlette `TemplateResponse` signature.** Starlette ≥ 0.36 requires `templates.TemplateResponse(request, name, context)`. The old `(name, context)` form raises `TypeError: unhashable type: 'dict'`. Do not put `"request": request` into the context dict — it's auto-injected now.
- **Vault git repo auto-init.** The wiki compiler calls `_ensure_git_repo()` on every run, which `git init`s the vault and creates an empty seed commit if no `.git` is present. `_git_commit()` also checks `git diff --cached --quiet` before committing to avoid empty commits.
- **Scheduler throttling shape.** The CLI backend is throttled by `max_calls_per_hour` (Max subscription limits are call-shaped, not token-shaped). The API backend uses `max_tokens_per_hour`. Both share `min_gap_seconds`. The scheduler targets sources in `TRANSCRIBED` state — earlier code mistakenly filtered on `ANALYZED`, making it a no-op.
- **Alembic + lovebug privilege gap.** `lovebug` owns the `second_brain` schema but does **not** have `CREATE` on the `petalbrain` database. `alembic/env.py` therefore guards `CREATE SCHEMA` behind a `pg_namespace` lookup, and the v1 migration intentionally drops the `IF NOT EXISTS` form (a bare `CREATE SCHEMA IF NOT EXISTS` raises permission denied even when the schema is already present). The schema is bootstrapped once, out-of-band, by `postgres` superuser.
- **public.embeddings is shared, not ours.** Alembic only manages the `second_brain` schema. The vector table itself is owned by `postgres` and shared across vault-mcp / ob1 / second-brain — we just INSERT into it. Rows are keyed by `(source_schema='second_brain', source_table='extractions', source_id, embedding_model='nomic-embed-text')`.
- **Embedding is best-effort.** The pipeline writes the relational `extractions` row first and `sess.flush()`es; only then calls `embed_extraction()`. Any failure (no DB URL, no Ollama, embedding 500) logs at WARNING and returns None — the relational row and file vault stay canonical.
- **File-based wiki, not petalbrain.wiki.** The wiki compiler still writes markdown into the configured `vault_path` and commits via git. We do **not** dual-write into `petalbrain.wiki` — that's vault-mcp's territory and intentionally out of scope.
## In-flight work
- **Vault location.** Recommended `/opt/projects/second-brain-vault/` as the production location (peer to the project dir). Not yet confirmed.
## Validation
Per Travis's global CLAUDE.md, run `zero-check` after any code change, and `a-review` after non-trivial diffs (> 50 lines). `a-review` script: `python3 ~/.claude/skills/a-review/run_review.py`, output goes to `reviews/a-review-<date>.md`.
Reference in New Issue View Git Blame Copy Permalink