Keyboard shortcuts

Press or to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Brain Server — API Contract (/recall + /ingest)

Wire contract for the brain-server HTTP API. The JSON shapes here are the source of truth; the Rust serde structs are kept equal to these shapes.

Status: /recall and /ingest are both implemented and live in the current source (see src/handlers/recall.rs, src/handlers/ingest.rs). They supersede the legacy /search and /ingest/markdown; the legacy endpoints remain for direct/CLI compatibility (documented in README.md and SPECS.md, out of scope here).

Versioning: the server reports SERVER_VERSION = env!("CARGO_PKG_VERSION") via /version and /health, and sets an X-Api-Version: <semver> response header on every route. Contract version: api v1.


Versioning & deprecation policy

Applies from v0.9.5 (“Inspect” M3) onward, before third parties depend on the API surface.

  • Version discovery. Every response carries X-Api-Version: <semver> (the crate version from Cargo.toml). Clients SHOULD log/record it; a major bump (1.x2.x) signals a breaking wire change.
  • Structured queries. The canonical query contract is the QueryDoc (see src/search/query.rs and openapi.yaml#/components/schemas/QueryDoc), sent to POST /recall. The legacy GET /search (flat q/lex/source) and POST /add remain functional but are deprecated.
  • Deprecation signal. Deprecated routes return an RFC 8594 Deprecation header (e.g. Deprecation: version="0.9.5"). The header names the version in which the route entered deprecation, not the version it will be removed. Removal only happens on a major-version boundary, and only after a minimum of one minor release of overlap with the replacement route.
  • Migration mapping.
    DeprecatedReplacement
    GET /search?q=...POST /recall with QueryDoc (structured lex, sources, intent, explain)
    POST /addPOST /ingest/memory (raw body) or POST /ingest/markdown (with title)
  • Stability promise. Within a major version, existing response shapes are additive (new optional fields only). A removed field or changed type is a breaking change and requires a major bump.

The full machine-readable route set lives in openapi.yaml (served at GET /openapi.yaml); keep the two in sync — the test_openapi_covers_routes unit test enforces it.


0. Conventions

ConcernRule
Content-Typeapplication/json (UTF-8) for all request/response bodies with a body
AuthAuthorization: Bearer <token> when a server-side token is configured (AUTH_TOKEN, or AUTH_TOKEN_FILE pointing at a 0600 file — the latter is preferred). Loopback may be exempt — server policy. Constant-time compare.
Unknown fieldsIgnored on deserialize (forward-compatible). Servers MUST NOT reject unknown keys.
Missing optional fieldsOmitted, not null. With exactOptionalPropertyTypes on the TS side, undefined keys are not serialized (conditional spread).
IDsKnowledge IDs are i64 (serialized as JSON number). Entity/relation IDs are not exposed over the wire by these endpoints.
StringsUTF-8; all bounds are UTF-8 byte lengths unless noted.
ErrorsUniform envelope (§5). Never leak internals (paths, SQL, stack).
TimeoutsServer enforces a 30 s per-request deadline + an 8 s /recall search budget; client also sets AbortController.

Field bounds (enforced server-side → 400 on violation)

FieldBoundError code
query1 ≤ len ≤ 2,000 (utf8 bytes)query_empty / query_too_long
limit1 ≤ n ≤ 100limit_out_of_range
title1 ≤ len ≤ 500title_invalid
content1 ≤ len ≤ 1,000,000 (1 MiB)content_empty / content_too_large
domainmatches ^[a-z0-9][a-z0-9_-]{0,62}$domain_invalid
entity/relation name1 ≤ len ≤ 100, ^[A-Za-z0-9 _-]+$name_invalid
entity typelen ≤ 64entity_invalid
relation type1 ≤ len ≤ 64, ^[a-z0-9_]+$ (snake_case)relation_invalid
arrays (entities/relations)≤ 200 each per requesttoo_many_entities / too_many_relations

Domain names are lowercase by convention. The server normalizes to lowercase (trim + lower) before comparison (so Healthhealth). Entity/relation names are also normalized to lowercase internally; their surrounding whitespace is collapsed. A well-formed but unregistered forced domain resolves to domain_invalid today (the domain_unknown distinction is reserved for a future per-domain registry; see §2).


1. Common types

Domain

A domain name string (see bounds above). The reserved domain global is the fallback sink.

Entity

{ "name": "vitamin d3", "type": "supplement" }
  • name — required, the entity surface form (case-insensitive unique within a domain).
  • type — optional free-form label (e.g. "supplement", "person", "concept").

Relation

{ "from": "vitamin d3", "to": "inflammation", "type": "helps" }
  • from/to — entity names (must match an Entity.name in the same payload OR an existing entity in the domain; server upserts entities as needed).
  • type — snake_case relation label.

RecallHit

{
  "id": 42,
  "title": "Vitamin D3 notes",
  "content": "Vitamin D3 supports immune function...",
  "score": 0.87,
  "domain": "health",
  "source": "both",
  "provenance": { "vector_rank": 0, "fts_rank": 1, "fused_score": 0.0327 }
}
FieldTypeAlways?Notes
idintegeryesknowledge id
titlestring | nullnoomitted if absent
contentstringyesthe matched chunk with a bounded, faithful snippet window
scorenumber (float)yesnormalized similarity/fusion score
domainstringnothe domain the hit came from (present when provenance=true)
source"vector" | "fts" | "both" | "graph"noretrieval path (present when provenance=true)
provenanceobjectnoper-retriever ranks + fused score (present when provenance=true)

provenance (per-hit)

The shape of RecallHit.provenance (defined in src/search/mod.rs):

FieldTypeNotes
vector_rankinteger | omittedrank the vector retriever assigned (0 = best)
fts_rankinteger | omittedrank the FTS5 retriever assigned
fused_scorenumber | omittedRRF-fused score
rerank_scorenumber | omittedcross-encoder score (only if the rerank tier ran)
rerank_truncatedbooleandoc was length-capped before reranking
prf_expandedbooleanhit surfaced via the PRF-expanded pass
top_retrieval_mode"vector" | "fts" | "both" | omittedwhich retriever(s) contributed the top result
retrieval_strategystring | omittedoverall strategy, e.g. hybrid or hybrid_prf
quality_assessmentobject | omittedheuristic confidence + recommendation (see src/search/quality.rs)
prf_decisionobject | omittedwhy PRF did/didn’t fire

2. POST /recall — deterministic recall

The server does everything: embed the query → auto-route via domain centroids → search (hybrid vec0 + FTS5, RRF fusion) → optional PRF query expansion → optional cross-encoder rerank → cross-domain fallback on miss → cap → return.

Request

{
  "query": "supplements for inflammation",
  "limit": 3,
  "domain": "health",        // optional: force a domain (disables auto-routing)
  "strict": false,           // optional: true = no cross-domain fallback
  "provenance": true,        // optional: include per-hit domain + source + provenance + telemetry
  // ── optional structured-query overrides (power tools) ──
  "source": "structured",    // filter: ingest kind, retrieval leg, or both (see table)
  "since": "2026-01-01",     // ISO-8601 / RFC3339; rows with created_at > since
  "lex": "inflammation -fever", // lexical (FTS5) query override
  "vec": "immune support",   // semantic embedding-query override
  "hyde": "Vitamin D3 reduces...", // hypothetical-answer embedding override (beats `vec`)
  "intent": "lookup"         // free-form intent label, recorded for provenance
}
FieldTypeRequiredDefaultNotes
querystringyesthe user turn / search text
limitintegerno5capped 1–100
domainstringno(auto-route)force a specific domain
strictbooleannofalsedisable fallback fan-out
provenancebooleannofalseinclude domain/source/provenance per hit + telemetry (domainsSearched is always present)
sourcestringnov1.13.3: an ingest kind (memory·markdown·structured·manual·vault) filters in SQL; a retrieval leg (vector·fts·graph) filters post-fusion; both is unrestricted. Unknown values return 422.
sourcesstring[]noOR filter over ingest kind (memory·markdown·structured·manual·vault) — filters the source column, NOT source URIs.
sincestringnoISO-8601 (RFC3339 or YYYY-MM-DD HH:MM:SS). Validated inside the search path; a malformed value is silently swallowed on the recall path today (the failing target contributes no hits) rather than surfacing a 400
lexstringnolexical (FTS5) query override (exact terms, phrases, -exclusions)
vecstringnosemantic embedding-query override
hydestringnohypothetical-answer embedding override; takes priority over vec
intentstringnofree-form intent label, recorded for provenance

Response — 200 OK

{
  "hits": [
    { "id": 42, "title": "Vitamin D3 notes", "content": "...", "score": 0.87, "domain": "health", "source": "both", "provenance": { "..." : "..." } },
    { "id": 88, "title": "Omega-3", "content": "...", "score": 0.71, "domain": "global", "source": "fts" }
  ],
  "domain": "health",
  "domainsSearched": ["health", "global"],
  "telemetry": { "embed_ms": 1.2, "vector_ms": 3.4, "fts_ms": 1.1, "fusion_ms": 0.1, "confidence": 0.78 }
}
FieldTypeAlways?Notes
hitsRecallHit[]yesordered by descending score; length ≤ limit
domainstringyesthe primary domain chosen by routing (or the forced domain)
domainsSearchedstring[]yesdomains of the returned hits (empty array when no hits). Always present (v1.13.3); no longer gated on provenance.
telemetryobjectnoper-stage retrieval telemetry. Present when provenance=true.

telemetry (per-response)

The shape of RecallResponse.telemetry (defined in src/search/mod.rs::SearchTelemetry):

FieldTypeNotes
embed_ms / vector_ms / fts_ms / fusion_ms / prf_ms / rerank_msnumberper-stage latency (ms)
retrieval_ms_vec / retrieval_ms_ftsnumberretrieval latency excluding embedding
vec_candidates / fts_candidates / fused_countintegercandidate counts before/after RRF
rrf_kintegerRRF k parameter (60)
confidencenumberheuristic quality-estimator score (0–1)
recommendationstring | omitted"return" / "run_prf" / "run_reranker" / "increase_top_k" / "clarify_query"
intent / embedding_querystring | omittedeffective intent / embedding query used

Routing semantics

  1. domain provided → search only that domain. Unknown/unresolvable → 400 domain_invalid.
  2. domain omitted (auto-route): a. Embed query once (model2vec). b. Compare to every domain centroid (int8/binary, Hamming/cosine). Rank domains. c. Primary domain = top centroid above DOMAIN_CONFIDENCE_THRESHOLD (0.55). d. If none above threshold → primary = global.
  3. Search the primary domain (hybrid vec0 KNN + FTS5 BM25, RRF fusion; optional PRF + rerank).
  4. Fallback (unless strict=true): if no confident route → fan out across all known domains + global; merge by score; tag each hit’s domain.
  5. Cap to limit; return.

Empty result is not an error — 200 with hits: [].

Errors

StatusCodeWhen
400query_empty / query_too_longmissing/oversized query
400query_rejectedquery matches a blocked prompt-injection pattern
400limit_out_of_rangelimit outside 1–100
400domain_invalidmalformed or unresolvable forced domain
401unauthorizedmissing/invalid bearer
429rate_limitedper-IP/domain rate limit breach
503recall_unavailablesearch task failed or exceeded the 8 s budget

domain_unknown is reserved for a future per-domain registry that distinguishes “well-formed but unregistered” from “malformed.” Today both resolve to domain_invalid.


3. POST /ingest — structured store (the KG write path)

Stores a knowledge entry + its embedding (auto-resolved domain if omitted), plus optional explicit entities/relations that populate the domain’s knowledge graph. The server trusts the caller’s graph data after validation (no server-side extraction — the annotation engine was retired in v0.9.0).

Request

{
  "title": "Vitamin D3 benefits",
  "content": "Vitamin D3 supports immune function and helps with inflammation...",
  "domain": "health",                          // optional: resolved domain if omitted
  "entities": [
    { "name": "vitamin d3", "type": "supplement" },
    { "name": "inflammation", "type": "condition" }
  ],
  "relations": [
    { "from": "vitamin d3", "to": "inflammation", "type": "helps" }
  ]
}
FieldTypeRequiredNotes
titlestringyes1–500 chars (trimmed)
contentstringyes1–1,000,000 chars (not trimmed)
domainstringnoforce domain; omit → resolved to global
entitiesEntity[]noupsert into the domain KG
relationsRelation[]noupsert; from/to upserted as entities if new

Response — 200 OK

{
  "id": 42,
  "status": "created",
  "domain": "health",
  "entitiesAdded": 2,
  "relationsAdded": 1
}
FieldTypeAlways?Notes
idintegeryesknowledge id. On duplicate, returns the existing knowledge id.
status"created" | "duplicate"yesduplicate = content_hash already present (xxh3-64 of content)
domainstringyesthe domain actually written to (forced or global)
entitiesAddedintegeryescount of entities in the request that were processed (upsert is idempotent, so this is the request count, not the delta of newly-inserted rows)
relationsAddedintegeryescount of relations in the request that were processed (same caveat)

Behavior

  • Dedup: content hashed (xxh3-64); exact dup → status: "duplicate", the existing id, no embedding work, no entity/relation mutation (entitiesAdded: 0, relationsAdded: 0).
  • Domain resolution: if domain omitted → resolved to global (no centroid routing on the write path today). After a successful write the server best-effort recomputes that domain’s centroid so future /recall auto-routing can target it.
  • Entities/relations are scoped to the resolved domain. INSERT OR IGNORE semantics (idempotent). from/to in relations[] are resolved to existing entity rows (they must already exist in entities[] or in the domain — relation insert fails if a referenced entity cannot be resolved).
  • Embedding: content is embedded once (model2vec) and stored in vec_knowledge as int8 + binary quantized vectors. The legacy f32 JSON embeddings column is no longer written.
  • Atomicity: knowledge + vec0 + entities + relations in one SQLite transaction.

Errors

StatusCodeWhen
400title_invalid / content_empty / content_too_largebounds violations
400name_invalidbad entity/relation name (empty, > 100, bad charset)
400entity_invalidentity type > 64 chars
400relation_invalidbad relation type (empty, > 64, not snake_case)
400too_many_entities / too_many_relationsarray > 200
400domain_invalidmalformed or unresolvable forced domain
401unauthorizedauth
413(bare status)body > 1 MiB (MAX_REQUEST_SIZE), enforced by the HTTP RequestBodyLimitLayer before the handler runs — returned as a plain 413, not the JSON envelope. (HandlerError::payload_too_large exists but is not invoked by this route.)
429rate_limitedper-IP/domain write limit
500internal_errorDB/embedding/transaction failure

4. Supporting endpoints

GET /health200

Illustrative example — the version is env!("CARGO_PKG_VERSION") at runtime and capacity is present only when the connection pool is not momentarily exhausted.

{
  "status": "ok",
  "version": "1.27.22",
  "model": "minishlab/potion-retrieval-32M",
  "system": { "memory_used_mb": 220, "memory_total_mb": 4096, "memory_percent": 5.4 },
  "pool":   { "connections": 2, "idle_connections": 1, "busy_connections": 1 },
  "backup": { "ok": true },
  "webhook": { "replay_secs": 600, "timestamp_required": 0, "scheme": "legacy" },
  "otel":   { "enabled": false, "endpoint": "http://127.0.0.1:4317" },
  "integrity": { "chain_ok": true, "last_checked_at": "...", "chain_head": "..." },
  "capacity": { "status": "ok", "docs": 430, "db_mib": 12, "rss_mib": 84 }
}

The primary consumer probes this to confirm the server is up (it only reads status). On failure the server returns { "status": "error", "version": "...", "error": "..." }. version is env!("CARGO_PKG_VERSION").

DELETE /memory/{id}200 / 404

{ "deleted": true }

Cascades to the entry’s vec_knowledge row (cleaned explicitly — vec0 has no FK), embeddings (FK CASCADE), and owned relations (FK SET NULL); the FTS trigger removes the FTS row. A tombstones row records the deletion for provenance. id is parsed as i64 (non-numeric → 400). 404 body: { "error": { "code": "not_found", "message": "..." } }.

GET /domains200 (ops/debug)

{
  "domains": [
    { "name": "global", "entries": 1307, "entities": 2341, "relations": 1892, "multi_db": false },
    { "name": "health", "entries": 412,  "entities": 2341, "relations": 1892, "multi_db": false }
  ]
}

Not used by the recall hot path, but useful for the brain CLI and for surfacing knownDomains.

v1.0.0 lifecycle routes (per the plan M5):

  • POST /domains body {"name": "health"} — create or warm a domain. Idempotent; returns 201 on first open, 200 if already present.
  • DELETE /domains/{name}?confirm={name} — drop ALL data for the domain and VACUUM. The global domain is protected. The ?confirm=<exact-name> query param is REQUIRED so a typoed URL or replay can’t destroy data by accident.
  • POST /domains/{name}/vacuum — reclaim free pages. Cheap, safe under load.
  • GET /domains/{name}/export — stream a consistent snapshot via VACUUM INTO. Returns application/octet-stream + Content-Disposition: attachment.
  • POST /domains/{name}/import — restore a snapshot into a NEW domain. Body is the raw bytes from a prior export. Target must not already exist; global is protected. Atomic temp-file + rename; migration runs on the imported pool.

Per-domain counts. In shim mode (BRAIN_MULTI_DB=false, the default) the registry enumerates the domain column on the shared pool — entities and relations are global totals in that mode. In multi-db mode each domain has its own file and the counts are genuinely domain-scoped.


5. Error envelope (uniform)

Every non-2xx response uses this shape:

{
  "error": {
    "code": "domain_invalid",
    "message": "domain 'heath' is not registered",
    "details": { "max": 200 }
  }
}
FieldTypeAlways?Notes
error.codestringyesmachine-readable snake_case code (see per-endpoint tables)
error.messagestringyessafe human text; never includes paths/SQL/secrets
error.detailsobjectnostructured context (e.g. {min, max} for range errors)

Consumers SHOULD treat any non-2xx as an error, distinguishing 404 from other statuses. 401 unauthorized MUST be surfaced (not silently swallowed) for security visibility.


6. Rust (Axum + serde) — canonical definitions

The shared response/error types live in src/handlers/mod.rs; the per-endpoint request types live alongside their handlers. Uses crates already in Cargo.toml (serde, serde_json, axum 0.8).

src/handlers/mod.rs — shared types

#![allow(unused)]
fn main() {
use axum::http::StatusCode;
use axum::response::IntoResponse;
use serde::{Serialize};
use serde_json::Value;

#[derive(Debug, Clone, Copy, Serialize, PartialEq, Eq)]
#[serde(rename_all = "lowercase")]
pub enum HitSource { Vector, Fts, Both, Graph }

#[derive(Debug, Serialize)]
pub struct RecallHit {
    pub id: i64,
    #[serde(skip_serializing_if = "Option::is_none")]
    pub title: Option<String>,
    pub content: String,
    pub score: f32,
    #[serde(skip_serializing_if = "Option::is_none")]
    pub domain: Option<String>,
    #[serde(skip_serializing_if = "Option::is_none")]
    pub source: Option<HitSource>,
    /// Per-retriever ranks + fused score. Present only when `provenance=true`.
    #[serde(skip_serializing_if = "Option::is_none")]
    pub provenance: Option<crate::search::Provenance>,
}

#[derive(Debug, Serialize)]
pub struct RecallResponse {
    pub hits: Vec<RecallHit>,
    #[serde(skip_serializing_if = "Option::is_none")]
    pub domain: Option<String>,
    /// v1.13.3 "SourceFix": always present (empty when no hits).
    pub domains_searched: Vec<String>,
    /// Per-stage retrieval telemetry. Present only when `provenance=true`.
    #[serde(skip_serializing_if = "Option::is_none")]
    pub telemetry: Option<crate::search::SearchTelemetry>,
}

#[derive(Debug, Serialize)]
pub struct IngestResponse {
    pub id: i64,
    pub status: &'static str, // "created" | "duplicate"
    #[serde(skip_serializing_if = "Option::is_none")]
    pub domain: Option<String>,
    #[serde(skip_serializing_if = "Option::is_none")]
    pub entities_added: Option<u32>,
    #[serde(skip_serializing_if = "Option::is_none")]
    pub relations_added: Option<u32>,
}

#[derive(Debug, Serialize)]
pub struct ForgetResponse { pub deleted: bool }

// ---------- uniform error envelope ----------

#[derive(Debug, Serialize)]
pub struct ErrorBody { pub error: ApiError }

#[derive(Debug, Serialize)]
pub struct ApiError {
    pub code: &'static str,
    pub message: String,
    #[serde(skip_serializing_if = "Option::is_none")]
    pub details: Option<Value>,
}

/// Handler error type → renders the uniform `ErrorBody` envelope.
#[derive(Debug)]
pub struct HandlerError { pub status: StatusCode, pub inner: ApiError }

impl IntoResponse for HandlerError {
    fn into_response(self) -> axum::response::Response {
        (self.status, axum::response::Json(ErrorBody { error: self.inner })).into_response()
    }
}
}

src/handlers/recall.rs — request

#![allow(unused)]
fn main() {
#[derive(Debug, Deserialize)]
pub struct RecallRequest {
    pub query: String,
    #[serde(default = "default_limit")]
    pub limit: u32,
    pub domain: Option<String>,
    #[serde(default)] pub strict: bool,
    #[serde(default)] pub provenance: bool,   // alias "explain"
    #[serde(default)] pub source: Option<String>,
    #[serde(default)] pub since: Option<String>,
    #[serde(default)] pub lex: Option<String>, // bare string or LexSpec object
    #[serde(default)] pub vec: Option<String>,
    #[serde(default)] pub hyde: Option<String>,
    #[serde(default)] pub intent: Option<String>,
    #[serde(default)] pub sources: Vec<String>, // OR filter over ingest kind
    #[serde(default)] pub profile: Option<String>,
    #[serde(default)] pub include_flagged: bool,
    #[serde(default)] pub as_of: Option<String>,
    #[serde(default)] pub evidence: bool,
    #[serde(default)] pub at: Option<String>,
    #[serde(default)] pub max_context_tokens: Option<usize>,
    #[serde(default)] pub gold_answer: Option<String>,
    #[serde(default)] pub graph: bool,
    #[serde(default)] pub include_decayed: bool,
    #[serde(default)] pub memory_kind: Option<String>,
    #[serde(default)] pub min_relevance: Option<String>,
    #[serde(default)] pub trace: bool,
}
}

Canonical field list as of v1.20.29; the current source is authoritative — see src/handlers/recall.rs.


### `src/handlers/ingest.rs` — request

```rust
#[derive(Debug, Deserialize)]
pub struct IngestRequest {
    pub title: String,
    pub content: String,
    pub domain: Option<String>,
    #[serde(default)] pub entities: Vec<EntityInput>,
    #[serde(default)] pub relations: Vec<RelationInput>,
}

#[derive(Debug, Deserialize)]
pub struct EntityInput {
    pub name: String,
    #[serde(rename = "type", default)]
    pub kind: Option<String>, // wire key is "type" (a Rust keyword)
}

#[derive(Debug, Deserialize)]
pub struct RelationInput {
    pub from: String,
    pub to: String,
    #[serde(rename = "type")]
    pub kind: String,
}

Validation constants & helpers (src/handlers/mod.rs)

#![allow(unused)]
fn main() {
pub const DOMAIN_RE: &str = r"^[a-z0-9][a-z0-9_-]{0,62}$";
pub const NAME_RE:   &str = r"^[A-Za-z0-9 _-]{1,100}$";
pub const RELTYPE_RE: &str = r"^[a-z0-9_]{1,64}$";

pub const MAX_QUERY: usize     = 2_000;
pub const MAX_TITLE: usize     = 500;
pub const MAX_CONTENT: usize   = 1_000_000;
pub const MIN_LIMIT: u32       = 1;
pub const MAX_LIMIT: u32       = 100;
pub const MAX_ENTITIES: usize  = 200;
pub const MAX_RELATIONS: usize = 200;
pub const MAX_BODY: usize      = 2 * 1024 * 1024; // 2 MiB — defined but UNUSED; real body cap is the HTTP layer (MAX_REQUEST_SIZE = 1 MiB)

pub const DEFAULT_RECALL_LIMIT: u32   = 5;
pub const DOMAIN_CONFIDENCE_THRESHOLD: f32 = 0.55;

pub fn normalize_domain(raw: &str)   -> Result<String, HandlerError>; // → domain_invalid
pub fn normalize_name(raw: &str)     -> Result<String, HandlerError>; // → name_invalid
pub fn normalize_rel_type(raw: &str) -> Result<String, HandlerError>; // → relation_invalid
}

provenance (src/search/mod.rs::Provenance) and telemetry (src/search/mod.rs::SearchTelemetry) are larger structs with nested quality-assessment and PRF-decision types (see §1 / §2 for their serialized field lists). Their full Rust definitions live in src/search/mod.rs and src/search/quality.rs.


7. JSON Schema generation (optional, future)

For a single machine-readable source of truth, derive JSON Schemas from the Rust structs via schemars (#[derive(JsonSchema)]) and publish them alongside the OpenAPI spec (openapi.yaml). The TS types can then be code-generated from those schemas, eliminating manual drift. Noted in ROADMAP Phase 6.


8. Capacity envelopes (v0.9.9)

brain-server publishes a measured (not estimated) capacity envelope per target hardware. A configuration that exceeds it is unsupported: writes are rejected with HTTP 507 Insufficient Storage until the operator resolves it; reads always return 200 (an over-capacity brain must still answer).

TargetBRAIN_CAPACITY_TARGETMax docsMax DBMax RSS
Jetson Nano 4 GB (default)jetson10 000512 MiB320 MB
Desktop / 16 GB hostdesktop50 0002 GiB320 MB
  • /health reports the live state under capacity: { target, docs, max_docs, db_mib, max_db_mib, rss_mib, max_rss_mib, status } where status is ok | warning (within 10% of a ceiling) | exceeded.
  • Writes (POST /add, /ingest, /ingest/memory, /ingest/markdown) call guard_capacity. Over-capacity → 507 with body { "error": "capacity_exceeded: docs=N/M db_mib=.../... rss_mib=.../..." }.
  • Reads (GET /search, POST /recall, GET /get/{id}) never check capacity — a brain over its envelope still answers queries.
  • Tightening for test/constrained deploys: CAPACITY_MAX_DOCS, CAPACITY_MAX_DB_MIB, CAPACITY_MAX_RSS_MIB override the built-in defaults.
  • Ship gate: bench --features bench with BENCH_ENVELOPE=jetson exits non-zero if RSS or p95 ceilings are breached — turning a measurement into an assertion.

Measured numbers for 1k / 10k / large-vault corpora are published in BENCHMARKS.md §v0.9.9 (operator step — run on the target hardware).

9. Migration (v0.9.9 — the v1.0 cutover contract)

v1.0.0 splits the single brain.db into per-domain files (global.db + brain-<domain>.db). v0.9.9 rehearses that cutover without performing it: the live runtime stays in shim mode (single global DB). The rehearsal proves the cutover is safe; v1.0.0 executes it.

Per-row migration rule

Every row follows exactly one rule when v1.0 runs the cutover:

Row kindDefault target domainRule
knowledge.domain = 'global'globalunchanged
knowledge.domain = '<name>'<name>copy to brain-<name>.db; tombstone in global
sources / source_revisionsfollows the linked chunk’s domaincopy with the chunks
entities / relationshipsfollows the owning knowledge.idcopy with the chunks
evidence_linksfollows from_chunk_idcopy with the from-chunk
tombstonesglobal (audit trail)never split
connectors / connector_checkpointsglobal (registry metadata)never split
audit_eventsglobal (immutable audit trail)never split
domain_centroidsglobal (it IS the routing table)never split
webhook_queueglobal (transient)drained before cutover; not migrated

Rehearsal tool

brain-migrate-rehearse (build with --features migrate) runs the cutover against a copy of the live DB:

# Stop the server first (WAL must be quiescent).
brain-migrate-rehearse rehearse \
  --source ~/.openclaw/workspace/brain.db \
  --dest   ~/.openclaw/workspace/global.db

Phases: backup (encrypted snapshot via backup::backup) → copy (VACUUM INTO + run_migration) → verify (row-count + content-hash + FTS/vec parity + source/revision linkage + evidence_links + audit_events + schema-version + 50-row vec0 byte spot-check) → report. Exits 0 only when every check passes; any mismatch leaves the dest file + a precise failure message. rollback removes the candidate without touching the source.

Recovery (rollback after the v1.0 cutover)

This is the procedure the rehearsal proves is safe:

  1. Stop the server.
  2. mv brain.db brain.db.pre-v1 and mv global.db brain.db (or flip BRAIN_DB_PATH).
  3. Enable BRAIN_MULTI_DB=true in the launchd plist.
  4. Restart via scripts/install-service.sh; brain doctor reports v1.0.0.
  5. Rollback if needed: stop server, mv brain.db.pre-v1 brain.db, unset BRAIN_MULTI_DB=true, restart. The failed global.db is retained as brain.db.failed-cutover for forensics.

v0.9.9 does NOT perform steps 1–5. It ships the tooling + this contract so v1.0.0 is a rehearsed operation.

v1.0.0 boot-time cutover (automatic)

When BRAIN_MULTI_DB=true is set at server startup, the server performs a one-shot safety snapshot of the legacy brain.db into global.db:

  1. Resolves paths via StorageLayout: legacy_db() (brain.db) and global_domain_db() (global.db).
  2. Skips the snapshot if ANY of:
    • shim mode (BRAIN_MULTI_DB off — the legacy brain.db IS the global pool);
    • the marker ~/.openclaw/workspace/.v1-legacy-cutover-done exists;
    • global.db already exists (operator provisioned it);
    • brain.db has no knowledge rows (fresh install).
  3. Otherwise: VACUUM INTO '<global.db>' (consistent snapshot, safe under WAL), then writes the marker so restarts never re-copy.

The runtime keeps reading the legacy brain.db for the global domain — the snapshot exists as a backup the rehearsal tool can verify against, and as the physical source for any future operator-driven cutover. No data is moved out of brain.db; the v0.9.x install path is preserved byte-identical.

v1.0 deprecation policy. The legacy /add, /search, /ingest/memory, and /ingest/markdown routes remain (with Deprecation: version="0.9.5" header). The primary write path is now POST /ingest; the primary read path is POST /recall. A future major version may remove the legacy routes after a deprecation window of at least one minor cycle.

/ingest/memory response (v1.13.3). POST /ingest/memory now returns real chunk ids: chunk_id (first inserted rowid, null when nothing added), chunk_ids (all inserted rowids), entries_added, duplicates_skipped, and status (success|unchanged|error). entry_id is retained as a deprecated alias of chunk_id (it previously held the count of entries added, not a usable id). similarity_score: 1.0 is kept as a legacy field.

15. UMP binding (v1.17.3) — Universal Memory Protocol 1.0

The Universal Memory Protocol is the open standard for portable AI agent memory: records carry content hashes and signatures, access is granted by capability tokens, and the same memory moves across servers, agents, and tools. This section is the exact binding brain-server implements. The UMP 1.0 surface is a bounded binding of the spec at github.com/edihasaj/universal-memory-protocol (SPEC.md, wire shape per the actual 1.0 spec, corrected in v1.17.2).

Levels (suite-verified against the reference runner; 13/13, UMP 1.0 / L3)

  • L0 — portable-record file binding: GET /export?format=ump|ump-md renders the existing export as UMP records; POST /ingest?format=ump|ump-md lowers them back (single record or a batch envelope {ump:"1.0", records:[…]}, per-record status, one failure does not abort).
  • L3 — local integrity layer: with an operator key configured (BRAIN_UMP_KEY_DIR, brain ump keygen), records carry the reference §2.8 integrity = {content_hash: "blake3:<base32>", signature: "ed25519:<std-base64>", signer: <did:key>} block (v1.17.4 shape — legacy v1.17.3 blocks still verify via dual-read); verify-on-read; capability tokens (§5.2) gate /ump/* + /export. Without a key the server degrades to L2 and GET /ump/capabilities reports conformance: "L2".

GET /ump/capabilities (also mounted as /.well-known/ump.json) is the §3.1 handshake: {server{name,version}, ump:"1.0", conformance, kinds, bindings:["http","mcp","file"], retrieval_signals, max_recall:50, writable:true, audit:true}.

Routes (non-public except capabilities//.well-known/ump.json)

RouteActionCapability verbNotes
POST /ump/rememberWritewrite (derive ok)§3.3 partial record → structured ingest; scope.owner must match principal or be absent
GET /ump/memory/{id}Readreadintegrity-verified on read; tampered → dropped
POST /ump/recallReadread§3.2 {results:[{record, score, signals{…}}]}; same retrieval core as /recall
POST /ump/reviseWritewrite (derive ok)patch → new chunk + supersession; {id, supersedes:[OLD]}
POST /ump/forgetWritewrite (derive ok)hard:false soft / hard:true purge; both tombstoned + audited
POST /ump/feedbackWritewrite (derive ok)outcome followed|overridden|ignored|contradicted → suggest-feedback upsert
GET /ump/subscribeReadreadSSE change feed; {kind,id} events only, never bodies
POST /ump/auditAdmin— (denied to tokens)§9 alias of /audit
GET /ump/audit/verifyAdmin— (denied to tokens)§9 alias of chain verify

Capability tokens (§5.2)

Compact alg.payload.sig (EdDSA) tokens {iss: did, verbs: [read|write|derive|export], scope:{project}, exp} signed by the operator key; accepted as Authorization: Bearer on /ump/* + /export. Verbs: reads need read, writes write or derive, export paths export. Scope must be absent/empty or "global". Expiry enforced at parse (middleware); verbs × scope at handler entry (cap_gate after authorize). Unknown/malformed/expired → unauthorized (401).

Redact semantics

exportable:false records are never emitted on non-owner/file paths; PII redaction ([redacted:…]) applies per the v1.14 principal rules on /ump/recall and /ump/memory/{id} reads.

§5.3 injection-resistant rehydration (documented obligations)

  • Server: verify-before-emit (integrity check before a record is returned) and scope/consent filter before ranking — both are already the recall pipeline order (verify on read; owner scope filter in the SQL).
  • Client (documented, not enforced): treat record bodies as untrusted data — structural framing only, never execute the body, never render markdown as a command channel. See SECURITY.md §UMP.