diff --git a/Sources/Dev/pbs-hub-data-import-and-mcp.md b/Sources/Dev/pbs-hub-data-import-and-mcp.md new file mode 100644 index 0000000..4923c4c --- /dev/null +++ b/Sources/Dev/pbs-hub-data-import-and-mcp.md @@ -0,0 +1,217 @@ +--- +created: '2026-05-19' +path: Sources/Dev +project: pbs-hub-data-import-and-mcp +tags: +- pbs +- pbs-hub +- mcp +- python +- flask +- automation +- json +- authelia +type: project-plan +--- + +## Goal + +Add two ways to populate pbs-hub project data from outside the GUI: a JSON paste-import affordance on the Overview surface, and a standalone MCP server (`pbs-hub-mcp`) that exposes the same upsert logic plus read/search tools to LLM clients. The driving use case is video planning sessions in LLM chats — get the plan into the system without retyping it. + +Both front doors share the same underlying upsert logic in pbs-hub's REST API. The JSON paste ships first (Phase 1), the MCP layers on top (Phase 2), and later phases add Authelia SSO auth and vector search. + +This plan depends on the schema delivered by `pbs-hub-scene-management` — `Project`, `Scene`, `SceneItem`, `ProjectMetadata`, and their defaults — being live before any phase here can ship. + +--- + +## Locked Decisions + +### Upsert behavior (shared across paste import and MCP) + +- **Additive upsert.** Import only adds and updates; never deletes. Records present in the system but missing from a payload are left alone. +- **Partial imports honored.** Missing keys in the payload mean "leave the existing value alone." Only fields present in the payload get touched. +- **Upsert keys:** + - Project — `slug`. If slug is present and matches, update that project. If present and no match, create a new project with that slug. If missing entirely (paste-import only, from within a project), populate the currently open project. + - Scene — `title` within the project. Renaming a scene creates a new record; the old one stays (additive). + - SceneItem — `label` within the scene. Same logic. +- **Cross-project warning** (paste-import only) — if the user is on Project A's Overview and pastes JSON whose slug matches a different project, show a confirmation before context-switching or creating. + +### JSON payload shape + +```json +{ + "slug": "blueberry-cobbler", + "title": "Blueberry Cobbler", + "outline": "...", + "treatment": "...", + "metadata": { + "logline": "...", + "working_title": "...", + "tagline": "..." + }, + "scenes": [ + { + "title": "hook", + "sort_order": 1, + "purpose": "...", + "in_shot": "...", + "background": "...", + "script": "...", + "items": [ + {"label": "blueberries", "item_type": "shot", "sort_order": 1}, + {"label": "jenny", "item_type": "shot", "sort_order": 2} + ] + } + ] +} +``` + +All top-level fields are optional except — for the paste-import-without-current-project case — `slug`. For the MCP `upsert_project` tool, `slug` is required. + +### MCP server design + +- **Standalone repo:** `pbs-hub-mcp` on GitHub. Independent lifecycle from pbs-hub. +- **Container:** Deployed on the pbs Linode, `/opt/docker/pbs-hub-mcp/`, managed via `wp-i` Ansible. Same docker network as pbs-hub. +- **Internal comms:** MCP server → pbs-hub REST API via docker network using existing X-API-Key auth. +- **External comms:** Traefik in front of MCP server for client connections. Two auth paths — bearer token straight through, otherwise Authelia SSO (Phase 3). +- **CI/CD:** Same shape as `pbsii-cicd-pipeline` — GitHub Actions, self-hosted runner on `ustest1`, staging Linode before prod. +- **Language:** Python + UV venv for dev, docker for deploy. Uses Anthropic's MCP Python SDK. + +### MCP tool surface (v1 — Phase 2) + +- **`upsert_project`** — additive upsert by slug. Creates if no match, updates if match. Same payload shape as paste-import. Returns resulting project state. +- **`list_projects`** — returns array of projects with summary fields (slug, title, status, type, last updated, scene count). Filters: `status`, `type`. Pagination: default 25 results, optional `limit` and `offset`. +- **`get_project`** — full project state by slug. Returns outline, treatment, metadata, all scenes, all scene items. +- **`search_projects`** — text search across project **titles only** for v1. Returns matching projects with summary fields. Vector search across other fields (outline, treatment, scripts, briefings, item labels) deferred to Phase 4. + +### Schema sync (MCP ↔ pbs-hub) + +- **Mode 1 — Hand-mirrored Pydantic models.** MCP repo has its own `models.py` mirroring pbs-hub's `Project`, `Scene`, `SceneItem` shapes by hand. +- **Pydantic `extra = "ignore"`** on MCP-side models so pbs-hub can add fields without immediately breaking the MCP. +- **Drift detection:** Round-trip integration test in CI — MCP tool creates a Project, fetches it back, asserts every field round-tripped. Runs against a real stood-up pbs-hub container (docker-compose or testcontainers). +- **Triggers to revisit and move to Mode 2 (shared schema package):** + - A production bug (not test-caught) traceable to schema drift + - A second non-CLI Python consumer of pbs-hub materializes + - Schemas grow past ~15 fields or 3+ new shared models added +- **If Mode 2 is needed later,** the path is: extract `pbs_manager/schemas/` as a pip-installable subdirectory, both repos depend on it via git URL pinning. ~1-day refactor. + +### Authentication + +- **Phase 2 (bearer token only).** MCP server validates `Authorization: Bearer ` header. Token stored in env file (root:root 0600, per env-hardening Phase 1 pattern). All four tools available to any client with the token. +- **Phase 3 (add Authelia SSO).** Traefik forward-auth to Authelia for clients without a bearer header. Two access paths coexist: bearer for headless clients (Claude Code), Authelia for browser clients (Claude.ai web). Depends on `pbs-security-hardening` deploying Authelia to the pbs Linode first. + +### Error semantics and observability + +- **Structured error responses with clear messages.** No silent failures. Errors include enough detail for the LLM (or human) to understand what went wrong. +- **Structured logging** of tool calls and outcomes. Enables debugging "the LLM said it saved my plan but the project doesn't show changes." + +### Scope discipline + +- pbs-hub-mcp is **video-production scoped**. Tools cover `projects`, `scenes`, `scene_items`, `project_metadata`. It does **not** touch `platform_posts`, the reel publishing flow, the comment-reply automation, recipes, or other PBS surfaces. +- Other PBS surfaces get their own future MCP servers (e.g., a future WPRM-MCP). Not this plan. + +--- + +## Open Items + +- [ ] **Bearer token rotation policy** — decide rotation cadence and process. Manual rotation, or do we want a small admin command for it? +- [ ] **Search ranking on v1 title search** — relevance order, alpha order, or last-updated-first? Decide before Phase 2 build. +- [ ] **`list_projects` default sort order** — last-updated-first probably, but confirm. +- [ ] **JSON validation error UX for paste import** — preview-with-errors view design needs Lovebug pass. Show what was parsed, highlight what failed, let user fix and retry without losing the paste. +- [ ] **MCP tool descriptions / hints** — the text shown to LLMs that describes each tool. Worth a careful pass to make them maximally useful for an LLM picking the right tool. + +--- + +## Phases + +### Phase 1 — JSON paste import (in pbs-hub) + +Add the upsert API endpoint and the paste-import GUI to pbs-hub itself. + +- [ ] Pydantic models for the JSON payload shape (`Project`, `Scene`, `SceneItem`, `ProjectMetadata`) — these become the source-of-truth that Phase 2 mirrors +- [ ] `POST /api/projects/import` endpoint accepting the JSON payload, applying additive upsert +- [ ] Upsert key resolution logic — slug for project, title within project for scenes, label within scene for items +- [ ] Cross-project warning logic — server returns a `409 Conflict` with project info when slug mismatches the project context; GUI handles the warning UX +- [ ] Import button on Overview surface +- [ ] Modal or page with textarea for paste, parse-and-preview view, confirm/cancel actions +- [ ] Error display — structured errors with field-level highlights when JSON is malformed +- [ ] Tests — round-trip create + update, partial update, slug-mismatch handling, malformed JSON + +### Phase 2 — pbs-hub-mcp server + +Standalone MCP server in its own repo, exposing the four tools, deployed to staging then prod. + +- [ ] New GitHub repo `pbs-hub-mcp` +- [ ] Project scaffold (UV, pyproject, MCP Python SDK) +- [ ] Hand-mirrored Pydantic models matching pbs-hub's payload shapes (with `extra = "ignore"`) +- [ ] HTTP client for pbs-hub REST API (X-API-Key from env) +- [ ] Tools implemented: + - [ ] `upsert_project` (wraps `POST /api/projects/import`) + - [ ] `list_projects` (wraps `GET /api/projects` with filters + pagination) + - [ ] `get_project` (wraps `GET /api/projects/`) + - [ ] `search_projects` (wraps `GET /api/projects/search?q=...&fields=title`) +- [ ] Structured logging — tool name, args summary, outcome +- [ ] Bearer token auth middleware +- [ ] Round-trip integration test in CI (creates project, fetches back, asserts shape) +- [ ] Dockerfile + docker-compose for Linode deploy +- [ ] Ansible config in `wp-i` for `/opt/docker/pbs-hub-mcp/` +- [ ] Traefik routing rules + bearer auth label +- [ ] GitHub Actions CI/CD matching `pbsii-cicd-pipeline` pattern (self-hosted runner on ustest1) +- [ ] Deploy to staging, verify all four tools work end-to-end from a real MCP client +- [ ] Deploy to prod +- [ ] Document tool descriptions / usage hints for LLM consumption (in repo README + tool docstrings) + +### Phase 3 — Authelia SSO auth layer + +Adds the browser-client SSO path. Depends on `pbs-security-hardening` deploying Authelia to the pbs Linode first. + +- [ ] Verify Authelia is live on pbs Linode (depends on `pbs-security-hardening`) +- [ ] Traefik forward-auth label on the MCP route, conditional on no bearer header +- [ ] Authelia access control rule for the MCP route — scoped to Travis's Google Workspace account +- [ ] Verify both auth paths work — bearer token still passes through, browser client gets SSO redirect +- [ ] Document the two access patterns and when each applies + +### Phase 4 — Vector search + +Replaces title-only search with semantic search across project content. + +- [ ] Decide vector store (pgvector on existing MySQL host? Separate Chroma/Qdrant container? Embeddings on disk?) — this is its own design conversation when Phase 4 starts +- [ ] Embedding generation pipeline — when project content changes, regenerate embeddings +- [ ] Fields embedded: outline, treatment, scene scripts, scene briefings, scene item labels, project metadata values +- [ ] `search_projects` tool extended to support semantic search with optional fallback to title-only +- [ ] Hybrid ranking (BM25 + vector) is a stretch goal, not a Phase 4 requirement + +### Phase 5+ — Future MCP surfaces (TBD) + +Things explicitly named for the future. Captured here so they don't get forgotten. Timing TBD as needs emerge. + +- [ ] Additional pbs-hub MCP tools (e.g., scene-level operations, item-level operations, project status transitions) +- [ ] WPRM MCP — separate server, recipe data domain (separate plan when scoped) +- [ ] Reels / `platform_posts` MCP — separate server (separate plan) +- [ ] Schema sync mode revisit — if Mode 1 triggers fire, refactor to Mode 2 (`pbs_manager/schemas/` extraction) +- [ ] MCP tool surface for paste-import preview-and-confirm workflow (so an LLM could call `preview_upsert` before `upsert_project` to show diff) + +--- + +## Notes + +### Cross-references + +- **Dependency:** `pbs-hub-scene-management` — delivers the underlying schema (`Project.outline`, `Project.treatment`, `Scene`, `SceneItem`, `ProjectMetadata`, etc.) that this plan upserts into. Phase 1 here cannot start until that plan's Phase 1 (schema migration) ships. +- **Future dependency:** `pbs-security-hardening` — Phase 3 (Authelia SSO) depends on Authelia being live on the pbs Linode. +- **Reference patterns:** + - `pbsii-cicd-pipeline` — CI/CD pattern this plan copies + - vault-mcp, trellis-mcp — existing standalone MCP servers in the homelab (note: both own their own data layer; pbs-hub-mcp is the first MCP that fronts a separate REST API) + +### Design notes + +- **MCP and paste are intentionally redundant.** Same upsert logic, two front doors. The MCP is the daily-use path in LLM chats. The paste is the fallback for when the MCP isn't connected, when working with exported text, or when debugging. Maintaining both has near-zero cost since they share the same backend endpoint. +- **Pattern-setting decision.** pbs-hub-mcp is the first homelab MCP that fronts a separate REST API in a different repo. Mode 1 (hand-mirrored) with explicit revisit triggers sets the pattern for future MCPs in the same shape (WPRM-MCP, etc.). Don't lock in a heavier pattern (Mode 2 shared package) before pain emerges. +- **Forcing function on `pbs-security-hardening`.** Phase 3 (Authelia) explicitly depends on Authelia being deployed to pbs Linode. The Phase 3 hook in this plan is a deliberate prompt to push `pbs-security-hardening` forward — Travis has been deferring it, and this gives it a concrete consumer. +- **No structured/markdown paste fallback.** JSON-only by design. LLMs are reliable at producing JSON when given a schema; markdown parsing was considered and rejected as adding ambiguity without much benefit. + +### Flagged for plan readers + +- **The JSON payload spec is the contract.** Both phases share it. Changes to it after Phase 1 ships need to coordinate with Phase 2's Pydantic mirror. +- **Bearer token leakage handling.** If a token is suspected leaked, rotate it in env, redeploy MCP container, update client configs. No automatic revocation today; manual is fine at this scale. +- **`pbs-hub-mcp` as the first cross-repo MCP** is worth documenting as the homelab's "API-fronting MCP" pattern reference once it's live. Future MCPs of the same shape (WPRM-MCP, etc.) should copy this structure. \ No newline at end of file