# embedding-chunking

Shared 512-token chunking for the OB1 / vault embedding stack.

Wraps `tiktoken` (`cl100k_base`) so all four embedders (`vault_mcp`,
`ob1-mcp`, `ob1-enricher`, `ob1-reflector`) split source text the same way
before passing to Ollama (`nomic-embed-text`, 768-d).

`cl100k_base` is not nomic's exact tokenizer but close enough for chunking
decisions, and far faster than round-tripping every token count through
Ollama.

## Use

```python
from embedding_chunking import chunk_text

for chunk in chunk_text(document):
    vec = embed_text(chunk.text)
    # upsert with chunk.index, chunk.text, chunk.token_count
```

Consumers must delete-before-insert keyed on `(source_schema,
source_table, source_id, embedding_model)` before re-embedding, otherwise a
shrinking source leaves orphan chunks that silently corrupt retrieval.

## Constants

- `CHUNK_SIZE = 512`
- `CHUNK_OVERLAP = 64`

Step is `CHUNK_SIZE - CHUNK_OVERLAP = 448`.