mcp: project-plan — PBS Hub — Data Import & MCP Server
This commit is contained in:
Lovebug MCP
parent
ec93dd28dc
commit
0e661bbf86
217
Sources/Dev/pbs-hub-data-import-and-mcp.md
Normal file
217
Sources/Dev/pbs-hub-data-import-and-mcp.md
Normal file
@ -0,0 +1,217 @@
|
|||||||
|
---
|
||||||
|
created: '2026-05-19'
|
||||||
|
path: Sources/Dev
|
||||||
|
project: pbs-hub-data-import-and-mcp
|
||||||
|
tags:
|
||||||
|
- pbs
|
||||||
|
- pbs-hub
|
||||||
|
- mcp
|
||||||
|
- python
|
||||||
|
- flask
|
||||||
|
- automation
|
||||||
|
- json
|
||||||
|
- authelia
|
||||||
|
type: project-plan
|
||||||
|
---
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
|
||||||
|
Add two ways to populate pbs-hub project data from outside the GUI: a JSON paste-import affordance on the Overview surface, and a standalone MCP server (`pbs-hub-mcp`) that exposes the same upsert logic plus read/search tools to LLM clients. The driving use case is video planning sessions in LLM chats — get the plan into the system without retyping it.
|
||||||
|
|
||||||
|
Both front doors share the same underlying upsert logic in pbs-hub's REST API. The JSON paste ships first (Phase 1), the MCP layers on top (Phase 2), and later phases add Authelia SSO auth and vector search.
|
||||||
|
|
||||||
|
This plan depends on the schema delivered by `pbs-hub-scene-management` — `Project`, `Scene`, `SceneItem`, `ProjectMetadata`, and their defaults — being live before any phase here can ship.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Locked Decisions
|
||||||
|
|
||||||
|
### Upsert behavior (shared across paste import and MCP)
|
||||||
|
|
||||||
|
- **Additive upsert.** Import only adds and updates; never deletes. Records present in the system but missing from a payload are left alone.
|
||||||
|
- **Partial imports honored.** Missing keys in the payload mean "leave the existing value alone." Only fields present in the payload get touched.
|
||||||
|
- **Upsert keys:**
|
||||||
|
- Project — `slug`. If slug is present and matches, update that project. If present and no match, create a new project with that slug. If missing entirely (paste-import only, from within a project), populate the currently open project.
|
||||||
|
- Scene — `title` within the project. Renaming a scene creates a new record; the old one stays (additive).
|
||||||
|
- SceneItem — `label` within the scene. Same logic.
|
||||||
|
- **Cross-project warning** (paste-import only) — if the user is on Project A's Overview and pastes JSON whose slug matches a different project, show a confirmation before context-switching or creating.
|
||||||
|
|
||||||
|
### JSON payload shape
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"slug": "blueberry-cobbler",
|
||||||
|
"title": "Blueberry Cobbler",
|
||||||
|
"outline": "...",
|
||||||
|
"treatment": "...",
|
||||||
|
"metadata": {
|
||||||
|
"logline": "...",
|
||||||
|
"working_title": "...",
|
||||||
|
"tagline": "..."
|
||||||
|
},
|
||||||
|
"scenes": [
|
||||||
|
{
|
||||||
|
"title": "hook",
|
||||||
|
"sort_order": 1,
|
||||||
|
"purpose": "...",
|
||||||
|
"in_shot": "...",
|
||||||
|
"background": "...",
|
||||||
|
"script": "...",
|
||||||
|
"items": [
|
||||||
|
{"label": "blueberries", "item_type": "shot", "sort_order": 1},
|
||||||
|
{"label": "jenny", "item_type": "shot", "sort_order": 2}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
All top-level fields are optional except — for the paste-import-without-current-project case — `slug`. For the MCP `upsert_project` tool, `slug` is required.
|
||||||
|
|
||||||
|
### MCP server design
|
||||||
|
|
||||||
|
- **Standalone repo:** `pbs-hub-mcp` on GitHub. Independent lifecycle from pbs-hub.
|
||||||
|
- **Container:** Deployed on the pbs Linode, `/opt/docker/pbs-hub-mcp/`, managed via `wp-i` Ansible. Same docker network as pbs-hub.
|
||||||
|
- **Internal comms:** MCP server → pbs-hub REST API via docker network using existing X-API-Key auth.
|
||||||
|
- **External comms:** Traefik in front of MCP server for client connections. Two auth paths — bearer token straight through, otherwise Authelia SSO (Phase 3).
|
||||||
|
- **CI/CD:** Same shape as `pbsii-cicd-pipeline` — GitHub Actions, self-hosted runner on `ustest1`, staging Linode before prod.
|
||||||
|
- **Language:** Python + UV venv for dev, docker for deploy. Uses Anthropic's MCP Python SDK.
|
||||||
|
|
||||||
|
### MCP tool surface (v1 — Phase 2)
|
||||||
|
|
||||||
|
- **`upsert_project`** — additive upsert by slug. Creates if no match, updates if match. Same payload shape as paste-import. Returns resulting project state.
|
||||||
|
- **`list_projects`** — returns array of projects with summary fields (slug, title, status, type, last updated, scene count). Filters: `status`, `type`. Pagination: default 25 results, optional `limit` and `offset`.
|
||||||
|
- **`get_project`** — full project state by slug. Returns outline, treatment, metadata, all scenes, all scene items.
|
||||||
|
- **`search_projects`** — text search across project **titles only** for v1. Returns matching projects with summary fields. Vector search across other fields (outline, treatment, scripts, briefings, item labels) deferred to Phase 4.
|
||||||
|
|
||||||
|
### Schema sync (MCP ↔ pbs-hub)
|
||||||
|
|
||||||
|
- **Mode 1 — Hand-mirrored Pydantic models.** MCP repo has its own `models.py` mirroring pbs-hub's `Project`, `Scene`, `SceneItem` shapes by hand.
|
||||||
|
- **Pydantic `extra = "ignore"`** on MCP-side models so pbs-hub can add fields without immediately breaking the MCP.
|
||||||
|
- **Drift detection:** Round-trip integration test in CI — MCP tool creates a Project, fetches it back, asserts every field round-tripped. Runs against a real stood-up pbs-hub container (docker-compose or testcontainers).
|
||||||
|
- **Triggers to revisit and move to Mode 2 (shared schema package):**
|
||||||
|
- A production bug (not test-caught) traceable to schema drift
|
||||||
|
- A second non-CLI Python consumer of pbs-hub materializes
|
||||||
|
- Schemas grow past ~15 fields or 3+ new shared models added
|
||||||
|
- **If Mode 2 is needed later,** the path is: extract `pbs_manager/schemas/` as a pip-installable subdirectory, both repos depend on it via git URL pinning. ~1-day refactor.
|
||||||
|
|
||||||
|
### Authentication
|
||||||
|
|
||||||
|
- **Phase 2 (bearer token only).** MCP server validates `Authorization: Bearer <token>` header. Token stored in env file (root:root 0600, per env-hardening Phase 1 pattern). All four tools available to any client with the token.
|
||||||
|
- **Phase 3 (add Authelia SSO).** Traefik forward-auth to Authelia for clients without a bearer header. Two access paths coexist: bearer for headless clients (Claude Code), Authelia for browser clients (Claude.ai web). Depends on `pbs-security-hardening` deploying Authelia to the pbs Linode first.
|
||||||
|
|
||||||
|
### Error semantics and observability
|
||||||
|
|
||||||
|
- **Structured error responses with clear messages.** No silent failures. Errors include enough detail for the LLM (or human) to understand what went wrong.
|
||||||
|
- **Structured logging** of tool calls and outcomes. Enables debugging "the LLM said it saved my plan but the project doesn't show changes."
|
||||||
|
|
||||||
|
### Scope discipline
|
||||||
|
|
||||||
|
- pbs-hub-mcp is **video-production scoped**. Tools cover `projects`, `scenes`, `scene_items`, `project_metadata`. It does **not** touch `platform_posts`, the reel publishing flow, the comment-reply automation, recipes, or other PBS surfaces.
|
||||||
|
- Other PBS surfaces get their own future MCP servers (e.g., a future WPRM-MCP). Not this plan.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Open Items
|
||||||
|
|
||||||
|
- [ ] **Bearer token rotation policy** — decide rotation cadence and process. Manual rotation, or do we want a small admin command for it?
|
||||||
|
- [ ] **Search ranking on v1 title search** — relevance order, alpha order, or last-updated-first? Decide before Phase 2 build.
|
||||||
|
- [ ] **`list_projects` default sort order** — last-updated-first probably, but confirm.
|
||||||
|
- [ ] **JSON validation error UX for paste import** — preview-with-errors view design needs Lovebug pass. Show what was parsed, highlight what failed, let user fix and retry without losing the paste.
|
||||||
|
- [ ] **MCP tool descriptions / hints** — the text shown to LLMs that describes each tool. Worth a careful pass to make them maximally useful for an LLM picking the right tool.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Phases
|
||||||
|
|
||||||
|
### Phase 1 — JSON paste import (in pbs-hub)
|
||||||
|
|
||||||
|
Add the upsert API endpoint and the paste-import GUI to pbs-hub itself.
|
||||||
|
|
||||||
|
- [ ] Pydantic models for the JSON payload shape (`Project`, `Scene`, `SceneItem`, `ProjectMetadata`) — these become the source-of-truth that Phase 2 mirrors
|
||||||
|
- [ ] `POST /api/projects/import` endpoint accepting the JSON payload, applying additive upsert
|
||||||
|
- [ ] Upsert key resolution logic — slug for project, title within project for scenes, label within scene for items
|
||||||
|
- [ ] Cross-project warning logic — server returns a `409 Conflict` with project info when slug mismatches the project context; GUI handles the warning UX
|
||||||
|
- [ ] Import button on Overview surface
|
||||||
|
- [ ] Modal or page with textarea for paste, parse-and-preview view, confirm/cancel actions
|
||||||
|
- [ ] Error display — structured errors with field-level highlights when JSON is malformed
|
||||||
|
- [ ] Tests — round-trip create + update, partial update, slug-mismatch handling, malformed JSON
|
||||||
|
|
||||||
|
### Phase 2 — pbs-hub-mcp server
|
||||||
|
|
||||||
|
Standalone MCP server in its own repo, exposing the four tools, deployed to staging then prod.
|
||||||
|
|
||||||
|
- [ ] New GitHub repo `pbs-hub-mcp`
|
||||||
|
- [ ] Project scaffold (UV, pyproject, MCP Python SDK)
|
||||||
|
- [ ] Hand-mirrored Pydantic models matching pbs-hub's payload shapes (with `extra = "ignore"`)
|
||||||
|
- [ ] HTTP client for pbs-hub REST API (X-API-Key from env)
|
||||||
|
- [ ] Tools implemented:
|
||||||
|
- [ ] `upsert_project` (wraps `POST /api/projects/import`)
|
||||||
|
- [ ] `list_projects` (wraps `GET /api/projects` with filters + pagination)
|
||||||
|
- [ ] `get_project` (wraps `GET /api/projects/<slug>`)
|
||||||
|
- [ ] `search_projects` (wraps `GET /api/projects/search?q=...&fields=title`)
|
||||||
|
- [ ] Structured logging — tool name, args summary, outcome
|
||||||
|
- [ ] Bearer token auth middleware
|
||||||
|
- [ ] Round-trip integration test in CI (creates project, fetches back, asserts shape)
|
||||||
|
- [ ] Dockerfile + docker-compose for Linode deploy
|
||||||
|
- [ ] Ansible config in `wp-i` for `/opt/docker/pbs-hub-mcp/`
|
||||||
|
- [ ] Traefik routing rules + bearer auth label
|
||||||
|
- [ ] GitHub Actions CI/CD matching `pbsii-cicd-pipeline` pattern (self-hosted runner on ustest1)
|
||||||
|
- [ ] Deploy to staging, verify all four tools work end-to-end from a real MCP client
|
||||||
|
- [ ] Deploy to prod
|
||||||
|
- [ ] Document tool descriptions / usage hints for LLM consumption (in repo README + tool docstrings)
|
||||||
|
|
||||||
|
### Phase 3 — Authelia SSO auth layer
|
||||||
|
|
||||||
|
Adds the browser-client SSO path. Depends on `pbs-security-hardening` deploying Authelia to the pbs Linode first.
|
||||||
|
|
||||||
|
- [ ] Verify Authelia is live on pbs Linode (depends on `pbs-security-hardening`)
|
||||||
|
- [ ] Traefik forward-auth label on the MCP route, conditional on no bearer header
|
||||||
|
- [ ] Authelia access control rule for the MCP route — scoped to Travis's Google Workspace account
|
||||||
|
- [ ] Verify both auth paths work — bearer token still passes through, browser client gets SSO redirect
|
||||||
|
- [ ] Document the two access patterns and when each applies
|
||||||
|
|
||||||
|
### Phase 4 — Vector search
|
||||||
|
|
||||||
|
Replaces title-only search with semantic search across project content.
|
||||||
|
|
||||||
|
- [ ] Decide vector store (pgvector on existing MySQL host? Separate Chroma/Qdrant container? Embeddings on disk?) — this is its own design conversation when Phase 4 starts
|
||||||
|
- [ ] Embedding generation pipeline — when project content changes, regenerate embeddings
|
||||||
|
- [ ] Fields embedded: outline, treatment, scene scripts, scene briefings, scene item labels, project metadata values
|
||||||
|
- [ ] `search_projects` tool extended to support semantic search with optional fallback to title-only
|
||||||
|
- [ ] Hybrid ranking (BM25 + vector) is a stretch goal, not a Phase 4 requirement
|
||||||
|
|
||||||
|
### Phase 5+ — Future MCP surfaces (TBD)
|
||||||
|
|
||||||
|
Things explicitly named for the future. Captured here so they don't get forgotten. Timing TBD as needs emerge.
|
||||||
|
|
||||||
|
- [ ] Additional pbs-hub MCP tools (e.g., scene-level operations, item-level operations, project status transitions)
|
||||||
|
- [ ] WPRM MCP — separate server, recipe data domain (separate plan when scoped)
|
||||||
|
- [ ] Reels / `platform_posts` MCP — separate server (separate plan)
|
||||||
|
- [ ] Schema sync mode revisit — if Mode 1 triggers fire, refactor to Mode 2 (`pbs_manager/schemas/` extraction)
|
||||||
|
- [ ] MCP tool surface for paste-import preview-and-confirm workflow (so an LLM could call `preview_upsert` before `upsert_project` to show diff)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
|
||||||
|
### Cross-references
|
||||||
|
|
||||||
|
- **Dependency:** `pbs-hub-scene-management` — delivers the underlying schema (`Project.outline`, `Project.treatment`, `Scene`, `SceneItem`, `ProjectMetadata`, etc.) that this plan upserts into. Phase 1 here cannot start until that plan's Phase 1 (schema migration) ships.
|
||||||
|
- **Future dependency:** `pbs-security-hardening` — Phase 3 (Authelia SSO) depends on Authelia being live on the pbs Linode.
|
||||||
|
- **Reference patterns:**
|
||||||
|
- `pbsii-cicd-pipeline` — CI/CD pattern this plan copies
|
||||||
|
- vault-mcp, trellis-mcp — existing standalone MCP servers in the homelab (note: both own their own data layer; pbs-hub-mcp is the first MCP that fronts a separate REST API)
|
||||||
|
|
||||||
|
### Design notes
|
||||||
|
|
||||||
|
- **MCP and paste are intentionally redundant.** Same upsert logic, two front doors. The MCP is the daily-use path in LLM chats. The paste is the fallback for when the MCP isn't connected, when working with exported text, or when debugging. Maintaining both has near-zero cost since they share the same backend endpoint.
|
||||||
|
- **Pattern-setting decision.** pbs-hub-mcp is the first homelab MCP that fronts a separate REST API in a different repo. Mode 1 (hand-mirrored) with explicit revisit triggers sets the pattern for future MCPs in the same shape (WPRM-MCP, etc.). Don't lock in a heavier pattern (Mode 2 shared package) before pain emerges.
|
||||||
|
- **Forcing function on `pbs-security-hardening`.** Phase 3 (Authelia) explicitly depends on Authelia being deployed to pbs Linode. The Phase 3 hook in this plan is a deliberate prompt to push `pbs-security-hardening` forward — Travis has been deferring it, and this gives it a concrete consumer.
|
||||||
|
- **No structured/markdown paste fallback.** JSON-only by design. LLMs are reliable at producing JSON when given a schema; markdown parsing was considered and rejected as adding ambiguity without much benefit.
|
||||||
|
|
||||||
|
### Flagged for plan readers
|
||||||
|
|
||||||
|
- **The JSON payload spec is the contract.** Both phases share it. Changes to it after Phase 1 ships need to coordinate with Phase 2's Pydantic mirror.
|
||||||
|
- **Bearer token leakage handling.** If a token is suspected leaked, rotate it in env, redeploy MCP container, update client configs. No automatic revocation today; manual is fine at this scale.
|
||||||
|
- **`pbs-hub-mcp` as the first cross-repo MCP** is worth documenting as the homelab's "API-fronting MCP" pattern reference once it's live. Future MCPs of the same shape (WPRM-MCP, etc.) should copy this structure.
|
||||||
Loading…
Reference in New Issue
Block a user