Vancetope — LLM Resource Management

Defines how LLM access is managed: providers, keys, quotas, routing. Core principle: Every LLM call runs under an Account. Credentials and quotas can be defined at any Scope level. See also: identity-credentials | architektur-scopes-clients | mcp-tool-routing

1. The Problem

The Brain makes LLM calls. However:

  • Different Accounts have different provider access
  • Different Projects can have their own budgets (e.g., research grant)
  • Some Accounts do not have their own key (they use a Project, Team, or Tenant key)
  • The Brain must instantiate the correct provider with the correct key for each call
  • Token consumption must be trackable per Account AND per Project

2. LLM Configuration at Each Scope Level

LLM providers, keys, and quotas can be defined at any level:

Level Example Use Case
Account Mike’s personal Claude key Personal use
Project Research project has 5M tokens/month from grant Budget per project
Team NLP team has a shared Anthropic key Shared access
Tenant Company-wide Google AI Key Default for all
Local Ollama on the server Free fallback

LLM-Config at the Account 

account:
  id: acc_mike
  llm_config:
    providers:
      - provider: anthropic
        model: claude-sonnet-4
        api_key_ref: cred_mike_anthropic
        priority: 1
      - provider: google
        model: gemini-2.5-pro
        api_key_ref: cred_mike_google
        priority: 2
    quota:
      daily_tokens: 500000
      monthly_tokens: 10000000
      max_tokens_per_call: 8192
    preferences:
      default_model: claude-sonnet-4
      planning_model: claude-sonnet-4
      execution_model: claude-sonnet-4
      light_model: gemini-2.5-flash

LLM-Config at the Project 

project:
  id: proj_transformer_review
  llm_config:
    providers:
      - provider: anthropic
        model: claude-sonnet-4
        api_key_ref: cred_proj_grant_anthropic   # Key from the research grant
        priority: 1
    quota:
      monthly_tokens: 5000000                     # 5M Tokens/month, grant budget
      max_tokens_per_call: 16384                  # Larger contexts for papers
    preferences:
      planning_model: claude-sonnet-4
      execution_model: claude-sonnet-4
      light_model: gemini-2.5-flash               # falls back to Account/Team/Tenant

LLM-Config at the Team 

team:
  id: team_nlp_research
  llm_config:
    providers:
      - provider: anthropic
        api_key_ref: cred_team_anthropic           # Team key
    quota:
      monthly_tokens: 20000000                     # 20M Team budget

LLM-Config at the Tenant 

tenant:
  id: tenant_acme
  llm_config:
    providers:
      - provider: google
        model: gemini-2.5-pro
        api_key_ref: cred_tenant_google            # Company key
        priority: 10
      - provider: local_ollama
        model: llama-3.3-70b
        endpoint: http://localhost:11434
        api_key: null
        priority: 99                                # last fallback
        quota: unlimited

3. Provider Resolution Cascade 

Account → Project → Team → Tenant → Local LLM → Error

When the Brain wants to make an LLM call:

1. Session has Account (acc_mike) + Project (proj_transformer_review)

2. Which model do we need?
   → Purpose (planning/execution/light) → Determine model

3. Find Provider + Key (Cascade):
   a. Does Account have provider for this model?      → use Account key
   b. Does Project have provider for this model?       → use Project key
   c. Does Team have provider for this model?          → use Team key
   d. Does Tenant have provider for this model?        → use Tenant key
   e. Can local LLM provide this model (or equivalent)? → use local
   f. Nothing? → Task failed

4. Check quota (ALL applicable levels):
   → Account quota OK?    (Mike's personal budget)
   → Project quota OK?    (Grant budget for this project)
   → Team quota OK?       (Team budget)
   → Tenant quota OK?     (Company budget)
   → ALL must be OK. One exceeded = blocked.

5. Execute call, track usage at all levels

Model Alias Resolution

Before the provider cascade takes effect, it must be clear which specific model is intended. Recipes (see recipes.md) and Engine defaults typically reference aliases instead of direct provider model strings — so that the same configuration works on any Tenant, regardless of which provider keys it has.

Implemented in AiModelResolver (vance-brain). Format: <prefix>:<rest>.

input := <prefix>:<rest>

  prefix ∈ AiModelService.listProviders() (gemini, anthropic, …)
    → Use DIRECTLY: (prefix, rest) as (provider, model)

  Setting `ai.alias.<prefix>.<rest>` is set
    → Resolve RECURSIVELY with the resolved value
       (Cycle detection + depth limit 8)

  prefix == "default" and no alias configured
    → Fallback: (ai.default.provider, ai.default.model)

  else
    → UnknownModelException

The default: namespace has the safety net fallback: as long as a Tenant has ai.default.provider + ai.default.model configured, bundled Recipes like default:fast, default:analyze work without further setup steps.

Example settings for a Tenant with a multi-provider setup:

acme:
  ai.default.provider:        { type: STRING, value: gemini }
  ai.default.model:           { type: STRING, value: gemini-2.5-flash }

  ai.alias.default.fast:      { type: STRING, value: gemini:gemini-2.5-flash }
  # chat = interactive chat interfaces (Arthur/Eddie/Discuss-Recipes):
  # the highest turn volume in the system — deliberately controllable
  # independently of the analyze tier. Unset = recipes fall back to analyze.
  ai.alias.default.chat:      { type: STRING, value: anthropic:claude-haiku-4-5 }
  ai.alias.default.analyze:   { type: STRING, value: anthropic:claude-sonnet-4-5 }
  ai.alias.default.deep:      { type: STRING, value: anthropic:claude-opus-4 }
  ai.alias.default.web:       { type: STRING, value: gemini:gemini-2.5-pro }
  ai.alias.default.code:      { type: STRING, value: anthropic:claude-sonnet-4-5 }

  # Optional: Fill-In-the-Middle-Completion for Follow-Up (Edit mode).
  # Unset = Chat path; if set, the model must have a `fimTemplate`
  # (model-quirks.yaml / per-model YAML), otherwise Fail-Closed.
  # See follow-up.md §5.
  ai.alias.default.fim:       { type: STRING, value: lmstudio:qwen3-coder-30b }

When does the mechanism operate relative to the §3 cascade: the alias resolver runs before provider resolution. First, the logical model label (e.g., default:analyze) is resolved to a concrete (provider, model), then §3 searches for the appropriate key in the Account/Project/Team/Tenant cascade. Aliases and quota cascade are orthogonal concepts: alias decides which model, cascade decides whose key.

Model size as tier hint: ai-models.yaml additionally declares size: SMALL|LARGE per model. This is used for Recipe prompt variant selection (see recipes.md §5.1). Aliases do not change the classification — the tier information always comes from the resolved model.

Comma Cascade Across Multiple Specs

The input of a model spec can be a list, separated by commas. The first element that is configured wins. Example in a Recipe:

params:
  model: default:arthur,default:chat,default:analyze

This reads as a degradation ladder, the first configured element wins: “take the engine-specific default:arthur if the Tenant has defined it — otherwise the interface tier default:chat — otherwise the strong default:analyze — otherwise (last element) the Tenant safety net ai.default.*.”

The chat rung: ai.alias.default.chat is the canonical tier for interactive chat interfaces — the arthur, eddie, and discuss Recipes carry the ladder default:<engine>,default:chat,default:analyze. Reason: the main chat is the interface with the highest turn volume in the system, and many turns are mini-turns; which model they deserve is a different question than “strongest reasoning”. Worker and internal Recipes (ford, LightLlm, Judges, Trillian-Loops) remain on analyze — these are work turns, not conversation. The analyze rung before the safety net keeps migration neutral: Tenants that do not set chat behave exactly as before.

Backward-Compat: Single-element (no comma) behaves identically to today — default:chat is syntactically a 1-element cascade.

Resolution per element (same rule set as above, but with different miss semantics):

  • Element is a direct <provider>:<model> → wins immediately.
  • Element is a named provider instance (ai.provider.<prefix>.type is set or _vance/model/<prefix>/_provider.yaml declares wireType) → wins immediately.
  • Element is an alias and ai.alias.<prefix>.<rest> is configured → recursively resolve with the resolved value (which itself can be a cascade).
  • Element is not configured → not the default: safety-net fallback as usual, but jump to the next cascade element.

Last-resort fallback only at the end: The last element of the cascade still falls back to the default: safety-net rule (ai.default.provider / ai.default.model). Thus, default:arthur,default:chat,default:analyze in the worst case (no alias of the ladder configured) lands on the Tenant default — does not throw an error.

Cascade also in alias values: The comma cascade applies wherever a spec string is resolved — including in the target of an alias. ai.alias.default.chat = anthropic:claude-haiku-4-5,openai:gpt-4o-mini is valid and acts as a two-stage cascade from the alias target. Cycle detection works per cascade element with its own seen set; the depth limit (MAX_DEPTH=8) remains unchanged.

When does the cascade continue: Today, exclusively on missing definition (no direct provider match, no instance type, no alias setting). Runtime errors (provider down, quota exhausted, model deactivated) do not abort the cascade and fail hard as before. Disable logic will later move into the same resolver method, so that all cascade consumers automatically benefit — without API changes.

Distinction from params.fallbackModels: fallbackModels (ChatBehaviorBuilder) is a runtime fallback chain at the provider level — the primary model is actually called, and only on quota/provider failure does the chain advance. The comma cascade is a setup-time cascade at the alias definition level — it selects the first configured model before the first API call. Both mechanisms are orthogonal and combinable: an entry in fallbackModels can itself be a comma cascade.

Example settings + Recipe:

# Tenant-Settings (acme)
ai.alias.default.arthur:  { type: STRING, value: anthropic:claude-opus-4 }
ai.alias.default.chat:    { type: STRING, value: gemini:gemini-2.5-flash }

# Recipe arthur (bundled)
arthur:
  engine: arthur
  params:
    model: default:arthur,default:chat,default:analyze

Tenant acme has default:arthur configured → arthur-Recipe uses claude-opus-4. Another Tenant without default:arthur, but with default:chat → gemini-2.5-flash; without both, but with default:analyze → claude-sonnet-4-5; without all three → ai.default.*. No one needs to create empty alias entries just for the Engine defaulting mechanism, and the degradation ladder only decides on configured rungs — the order in the Recipe is the evaluated preference.

Whitespace and empty elements: Whitespace around comma separators is trimmed (a , b ≡ a,b). Empty elements (a,,b) are skipped.

Model metadata — Catalog storage: Model metadata (Context-Window, Default-Max-Output, size, capabilities, stripThinkTags, Pricing) is not stored in a monolithic file, but lives as a separate document per model under two parallel paths: _vance/model/<provider>/<slug>.yaml (Operator-managed) and _vance/model-auto/<provider>/<slug>.yaml (Discovery-Service-managed). Storage, cascade, cache strategy, and Discovery Service are described in §3a.

Named Provider Instances

The left part of a model spec (<prefix>:<model>) is semantically a provider instance, not necessarily the protocol type. Default: instance name == ProviderType.wireName() (openai, anthropic, …) — thus all existing specs and settings remain valid.

Tenants can define additional named instances to, for example, use multiple OpenAI-compatible endpoints (real OpenAI, DeepSeek-Direct, OpenRouter, local vLLM) in parallel — each with its own apiKey and baseUrl, but the same wire protocol:

acme:
  # Standard instance "openai" → real OpenAI
  ai.provider.openai.apiKey:           { type: PASSWORD, value: sk-... }

  # Additional instance "deepseek-direct" → OpenAI-Wire, but DeepSeek-Endpoint
  ai.provider.deepseek-direct.type:    { type: STRING,   value: openai }
  ai.provider.deepseek-direct.baseUrl: { type: STRING,   value: https://api.deepseek.com/v1 }
  ai.provider.deepseek-direct.apiKey:  { type: PASSWORD, value: sk-... }

  # Alias references the instance directly
  ai.alias.default.analyze:            { type: STRING,   value: deepseek-direct:deepseek-v4-flash }

Resolution order in AiModelResolver (extended):

input := <prefix>:<rest>

  prefix ∈ ProviderType.wireName() (openai, anthropic, …)
    → (protocolType=prefix, instance=prefix, model=rest)

  Setting `ai.provider.<prefix>.type` is set
    → instance recognized; (protocolType=<value>, instance=prefix, model=rest)
       Unknown type-wireName → UnknownModelException

  Document `_vance/model/<prefix>/_provider.yaml` declares `wireType`
    → instance recognized; (protocolType=<wireType>, instance=prefix, model=rest)
       Unknown wireType → UnknownModelException (names the document)

  Setting `ai.alias.<prefix>.<rest>` is set
    → recursively resolve with the resolved value

  prefix == "default" and no alias
    → Fallback (ai.default.provider, ai.default.model)

  else
    → UnknownModelException

What the instance selects:

  • AiChatConfig.providerInstance decides under which path ai.provider.<instance>.{apiKey,baseUrl} is read.
  • ModelCatalog indexes its YAML sections by instance name. A named instance carries its own metadata section: 
    deepseek-direct:
  deepseek-v4-flash:
    contextWindowTokens: 1048576
    size: SMALL
  • AiChatConfig.providerType continues to provide the protocol for adapter dispatch in AiModelService (which AiModelProvider bean builds the chat).

What the instance does not change: the wire model name that goes to the API (rest part). To use the same model designation with different metadata configs, create two instances with the same type that differ per instance under _vance/model/<instance>/<modelName>.yaml (see §3a).

Protocol from the Catalog: _provider.yaml → wireType

The protocol can also be declared by the provider sidecar instead of via setting: _vance/model/<instance>/_provider.yaml with wireType: openai. Both ways bind the same thing; the setting wins where it is set (per-Tenant override), the document is the supplied default.

The reason is a measured failure mode, not convenience: whoever creates a model directory has already stated which protocol the endpoint speaks. If the same statement had to be in a setting a second time, that was precisely the step that was omitted — and the failure was neither explanatory nor in the right place: the resolution failed on the alias branch and reported “alias not configured”, i.e., neither the missing key nor the directory in question.

Two related consequences:

  • Visibility follows the catalog snapshot. A sidecar written at runtime only takes effect after the next refresh (30 min or POST /brain/{tenant}/admin/ai-models/refresh) — the same latency as for a newly created model document. A setting takes effect immediately.
  • Discovery reads the same order. ModelDiscoveryService otherwise derives the protocol from the instance name; an instance declared this way would fail ProviderType.fromWireName and be silently skipped. An instance that the resolver can chat with, but whose “Discover AI Models” button does nothing, is a state that no one can explain from the outside.

Exactly one instance of this type is supplied: cortecs (wireType: openai, without model documents — the catalog comes from Discovery). It is also the example for “add your own gateway”: create sidecar, copy credential form (see below), done.

Credentials: one setting form per instance

Access data is in one form per provider instance (_vance/setting_forms/llm-provider-<instance>.yaml), not in a common LLM form. This is not a matter of taste: bindsTo.key is a fixed string, so a form can address exactly one instance. As long as the credentials were in a shared form, a second OpenAI-wire instance could simply not be configured via the interface — the only way offered was to redirect ai.provider.openai.baseUrl, which also takes over the key and endpoint of the real OpenAI instance.

Supplied: anthropic, openai, openai-experimental, gemini, ollama, lmstudio, cortecs. Keyless providers (Ollama, LM Studio) have no key field — ProviderType.requiresApiKey() is false there, the field would not be read by anyone. Anthropic and Gemini have no baseUrl field: their adapters read a base URL only for the model listing call, not for chat.

llm-setup.yaml retains what is truly global: aliases, default pair, embeddings, tracing.

Scope Pinning: params.aiScope

Alias, ai.default.*, ai.provider.<instance>.apiKey, .baseUrl, and the ModelCatalog view are separate cascade lookups (think-process → project → _tenant). A project that only overrides a part of this mixes layers: if it sets ai.provider.openai.baseUrl to a different endpoint but inherits the model name from _tenant, the Tenant model goes to the Project endpoint (symptom: 404 “model does not exist” despite valid config at both levels).

A Recipe can therefore pin its AI config to the outermost layer via a parameter:

params:
  aiScope: tenant     # Default: cascade

tenant means: all mentioned lookups run with projectId=null/processId=null, so the cascade collapses to _tenant. Model and endpoint thus come from the same level by design. There is deliberately no fallback to the Project level if the Tenant has configured nothing — “sometimes Tenant, sometimes Project, depending on which key is set” would be precisely the non-determinism that pinning eliminates. The pinned Engine would then fail (for best-effort services, this means no execution).

The criterion for pinning is not “service engine”, but is the output control data for others: Agrajag marks tools UNAVAILABLE and sets cooldowns that slow down other processes — this decision must not depend on a project’s experimental model. User-facing helpers (how_do_i, follow-up) deliberately remain on cascade and follow the project model. The process itself remains in its project; only the AI config (AiConfigScope, read in ChatBehaviorBuilder/EngineChatFactory) is pinned.

3a. Model Catalog Storage — Per-Model Documents

Model metadata is not kept in a monolithic ai-models.yaml. Each model is a separate document — split into two path prefixes per scope: _vance/model/** (Operator-managed, “manual”) and _vance/model-auto/** (ModelDiscoveryService-managed, “auto”). This makes single-model overrides trivial, avoids merge conflicts in a single file, and makes the Discovery Service (§3a.6) safe against Operator edits.

3a.1 Path Convention

The catalog knows two parallel path prefixes per scope — deliberately separated by path and not by flag, so that auto-writes can never overwrite manual data and vice versa:

_vance/model/<providerInstance>/<filenameSlug>.yaml          ← MANUAL  (Operator/Maintainer)
_vance/model-auto/<providerInstance>/<filenameSlug>.yaml     ← AUTO    (Discovery Job, §3a.6)
_vance/model/<providerInstance>/<sub>/<filenameSlug>.yaml    ← nested for '/' in wire model name
Path Prefix Owner Provenance Who Writes
_vance/model/** Operator / Maintainer Hand-maintained; carries pricing, capabilities, custom overrides UI/Setting Forms, Eddie via manual_read('ai-model-catalog'), direct Doc edits
_vance/model-auto/** Automation Provider listing output; only what the vendor API provides ModelDiscoveryService (§3a.6); never manual

Both paths exist in every scope (Project, _tenant, _vance). Within a scope, manual is applied after auto — so manual wins field-wise (see §3a.4). Discovery is free to overwrite within its own subtree half.

Naming Convention (applies identically to both prefixes):

  • providerInstance — Instance name from §3 (Default: ProviderType.wireName() like anthropic, openai, gemini, ollama, lmstudio, ollama-cloud; or named like deepseek-direct). Directory name must match [a-z0-9._-]+.
  • filenameSlug — Filename without .yaml. Must match [A-Za-z0-9._-]+. Subdirectories under the provider directory carry the / part of a wire model name (see next point).
  • Wire model name (what goes to the provider API):
    • Default: relative path under the provider directory, without .yaml extension. Example: _vance/model/lmstudio/mlx-community/Qwen3.6-35B-A3B-4bit.yaml → Wire name mlx-community/Qwen3.6-35B-A3B-4bit.
    • Override via YAML field wireName: ... — for model names with : (Ollama tags like qwen3:30b) or other filename-unsafe characters. Example: _vance/model/ollama/qwen3-30b.yaml with wireName: "qwen3:30b".
  • Provider sidecar for endpoint facts, not for credentials. An _provider.yaml may exist per provider directory. It carries what applies to all models behind the same endpoint and does not need to be repeated per model: wireType (the protocol — binding, see §3 “Protocol from the Catalog”), maxTools (the endpoint’s tool array limit), and display metadata (displayName, authType). Credentials remain settings (ai.provider.<instance>.{apiKey,baseUrl}) — a secret in a catalog document would be included in every Kit export and every doc_read.
  • Validation happens during catalog build. Invalid provider names or file slugs → Skip + WARN log, so a single typo does not block the entire catalog.

3a.2 Model Document — YAML Schema

Identical schema for MANUAL and AUTO paths — what differs is only which fields are typically set.

# _vance/model/anthropic/claude-sonnet-4-6.yaml         (Example manual)
wireName: "claude-sonnet-4-6"    # only if different from path (e.g., Ollama tags)
contextWindowTokens: 200000
defaultMaxOutputTokens: 8192
size: LARGE                      # SMALL | LARGE — Recipe tier hint
kind: chat                       # chat | image (default chat)
capabilities:                    # List, replaced as a whole
  - vision
  - pdf
  - thinking
stripThinkTags: false            # default false — see §4.1
messageParser: null              # optional — Name of a registered MessageParser bean
                                 # (gemma4, deepseek-v4, …). Falls back to model-quirks.yaml
                                 # pattern match, see §4.1.1.
outputTokenParam: max_tokens     # optional — OpenAI-Wire field for the output cap
                                 # (max_tokens | max_completion_tokens), see §4.1.2
fimTemplate: null               # optional — Fill-In-the-Middle-Prompt-Shape of a
                                 # completion-trained model, e.g.,
                                 # "<fim_prefix>{prefix}<fim_suffix>{suffix}<fim_middle>".
                                 # Family-specific (Qwen/StarCoder, DeepSeek, Codestral
                                 # differ); falls back to model-quirks.yaml
                                 # pattern match. Consumer: FimCompletionService
                                 # (follow-up §5), Gate: ai.alias.default.fim
pricing:                         # Operator value — wins per field; discovery only writes auto:true files
  currency: USD
  inputPerMTok: 3.00
  outputPerMTok: 15.00
  cacheReadPerMTok: 0.30
  cacheWritePerMTok: 3.75
discoveredBy: manual             # "manual" in manual path, "discovery-job" in auto path
discoveredAt: "2026-06-27T10:00:00Z"

Fields typically per layer:

Field Bundled (manual) _tenant manual _tenant auto project manual project auto
contextWindowTokens yes on override on override (Gemini provides it) on override on override
pricing yes (Vendor standard) yes (Enterprise deals) never yes never
capabilities yes on override never (API listing doesn’t know this) on override never
kind yes on override if listing provides it on override if listing provides it
stripThinkTags yes (Reasoning models) on override never on override never
messageParser rarely (only if pattern in model-quirks.yaml is insufficient) on override never on override never
fimTemplate rarely (only if pattern in model-quirks.yaml is insufficient) on override never on override never

Required fields: contextWindowTokens, size (effectively via cascade resolve — an auto-doc alone doesn’t need them as long as bundled/manual provides them). Everything else is optional. pricing null means “unpriced” for cost tracking: the call still lands in the ledger (tokens, calls, model, caller), only the cost columns remain 0 and currency null. An unpriced model must never look like an unused one in usage statistics.

Missing pricing means “price unknown”, not “free” — the daily bucket counts such calls in unpricedCalls and the report shows its coverage, instead of silently adding a zero to the sum. To keep this clear, locally running models (ollama/lmstudio) carry an explicit zero rate (inputPerMTok: 0.0) instead of no block at all.

defaultMaxOutputTokens for reasoning models. On the OpenAI wire, reasoning_content tokens count against max_tokens. A cap measured only for the visible response is therefore eaten up by the Thinking pass: the provider responds with HTTP 200, finish_reason: "length" and completely empty content — no text, no tool call. For every reasoning-capable model, the cap must therefore cover Thinking plus response (bundled value for GLM-/DeepSeek families: 32768). The resilience layer treats precisely this combination (empty + finish=LENGTH) as not retryable — an identical re-request hits the same wall — and passes the finishReason to the Engine so that the user message says “Output limit reached” instead of “transient provider glitch” (see §8).

Provider Document — _provider.yaml. Optional per provider directory, a document next to the model files, merged according to the same cascade rules:

displayName: Cortecs (OpenAI-wire gateway)
wireType: openai        # Protocol of the instance — see §3
authType: api-key
maxTools: 128           # Endpoint limit, not model limit
tlsInsecure: true       # optional: endpoint behind internal CA — Trust-all-TLS for Chat + Listing

tlsInsecure — Mitigation for endpoints behind an internal CA. A gateway with a certificate on a corporate root CA fails every chat and listing call with PKIX path building failed — the JVM truststore does not know the CA. The operator-side fix is importing the CA into the JVM truststore (global, restart-bound); the per-instance fix is tlsInsecure: true in the sidecar: chat calls (AiChatConfig.insecureTls, via the resolver from the same sidecar lookup as wireType) and Discovery listing (ProviderListingRequest.insecureTls) for this instance then run via a trust-all-SSLContext (TlsInsecure). Fail-closed: if the field is missing or not explicitly true, validation remains active. Deliberately an opt-in per instance with ADMIN write gate on the _vance/ document — certificate validation (including hostname and chain) is disabled for exactly one named endpoint family; embedding endpoints are not covered.

The dividing line to settings is not “metadata here, config there”, but who owns the statement: wireType/maxTools are endpoint properties that are the same for every Tenant and can therefore be supplied; apiKey/baseUrl belong to the Tenant and remain settings. Therefore, the type setting overrides the wireType of the document — a Tenant may redirect a supplied assignment, but not vice versa.

3a.3 Override Cascade

ModelCatalog merges seven layers per (providerInstance, modelName) — three scope levels × {auto, manual} plus Bundled as the base. Inner wins, manual wins within a scope over auto:

project-manual  ← innermost, beats everything
project-auto
_tenant-manual
_tenant-auto
_vance-manual   (system tenant, global Maintainer layer)
_vance-auto     (typically empty — Discovery runs per-Tenant)
bundled         ← outermost (classpath)

Pseudo-code of the application order (outer → inner; each layer overwrites fields it sets):

apply bundled
apply (_vance, _tenant) auto      then manual
if tenant given:
  apply (tenant, _tenant) auto    then manual
  if project != _tenant:
    apply (tenant, project) auto  then manual
  • Merge is deep, per field. An override only provides the fields it changes; unset fields are inherited.
  • Lists are replaced as a whole (especially capabilities, supportedAspectRatios) — so that owners can both add and remove individual values. Concatenation semantics would be ambiguous.
  • Manual over Auto per Scope. Within a (tenant, project) scope, the manual layer is applied after the auto layer — so manual hand-edits win field-wise against fresh Discovery data at the same scope. Discovery is free to overwrite in its subtree (_vance/model-auto/**) without losing manual overrides.
  • Inner-Scope-Auto beats Outer-Scope-Manual. Example: (tenant, project) auto sets contextWindowTokens = 50000; (tenant, _tenant) manual has 100000. Lookup at (tenant, project) yields 50000 — Project is the inner scope. This is intentional: Project-specific reality beats Tenant default, regardless of who wrote it.
  • Bundled layer is located under vance-brain/src/main/resources/vance-defaults/_vance/model/<providerInstance>/<filenameSlug>.yaml (path mirror in the classpath, same convention as DocumentService.RESOURCE_PREFIX = "vance-defaults/"). There is no bundled layer for model-auto/** — Bundled is always manual-shaped. With a new Brain version, bundled updates are not automatically mirrored in Tenant docs.

Engines automatically pass tenantId/projectId from the Process via AiChatOptions to the per-call providers, so that their capability lookups (Vision/PDF packaging) also see the scope-specific view.

3a.4 Cache — Atomic-Swap Refresh

ModelCatalog keeps Bundled + Per-Scope layers (manual and auto separately) in memory as an immutable Snapshot. Lookup is O(1) and in the hot path of every LLM call; a memoized merged view per (tenantId, projectId) is built lazily on first access and cached in the snapshot.

Initial Load on boot:

  1. Classpath scan over vance-defaults/_vance/model/**/*.yaml → Bundled layer (manual-shaped).
  2. findAllByPathPrefix("_vance/model/") → a map (tenantId, projectId) → Manual layer.
  3. findAllByPathPrefix("_vance/model-auto/") → a map (tenantId, projectId) → Auto layer.
  4. Build a complete new Snapshot in a local variable, then assign it to the active cache pointer with one atomic volatile write.

Refresh (every 30 minutes, scheduled): identical loader — new snapshot built completely, then atomic swap. No partial updates to the running cache, so readers never see an inconsistent intermediate state.

Refresh on Demand:

  • REST: POST /brain/{tenant}/admin/ai-models/refresh (Admin right via RequestAuthority.enforce(Tenant, ADMIN)). Body empty. Response: { refreshedAt, bundledModelsLoaded, bundledProvidersLoaded, overrideScopes, durationMs }. Synchronous — response only comes after swap.
  • UI: Button in the Profile Editor (Actions section) calls the same endpoint and shows the counters as a toast.

Deliberately no DocumentChangedEvent invalidation. Catalog contents change rarely (models per Tenant in the order of dozens, update frequency days to weeks). A 30-min sliding freshness plus explicit trigger is enough — and avoids a listener path that would have to react to every settings/wizard write.

Pod Locality: Each pod has its own cache. Refresh is not coordinated — in multi-pod setups, a refresh drifts by a maximum of 30 minutes between pods, which is acceptable for model catalog data. For cluster-wide immediate consistency, call the REST endpoint per pod (or via a cluster broadcast, which is not part of v1).

3a.5 Bootstrap & Migration

First-Boot Bootstrap: A ModelCatalogBootstrapper runs once at Brain startup. If the _vance Tenant contains no documents under _vance/model/**, it copies all Bundled files there. This makes the initial Tenant productive without anyone having to manually write YAML beforehand. Later Brain versions execute the bootstrap again, but only add missing models — existing Tenant edits are not overwritten. _vance/model-auto/ is never populated by the bootstrapper — that is Discovery domain.

No migration path from old monolithic ai-models.yaml. The old format is replaced with this spec — Brain reads neither _vance/ai-models.yaml, _tenant/ai-models.yaml, nor Project ai-models.yaml. Existing bundled ai-models.yaml was converted to the new directory structure during the build (one-time code change, no runtime fallback).

3a.6 Discovery Service

ModelDiscoveryService populates the auto layer (_vance/model-auto/**) deterministically from the vendor listing APIs. Operator edits under _vance/model/** are never touched — the two layers are physically separated by the path prefix.

Trigger:

  • REST POST /brain/{tenant}/admin/ai-models/discover (Admin right). Synchronous, returns counters.
  • UI: “Discover AI Models” button in the Profile Editor next to “Refresh”.
  • (Optional) UrsaScheduler Recipe for scheduled runs (Default off).

Scope Symmetry: Discovery reads provider credentials non-cascaded per (tenant, project) and writes the auto-docs to the same scope. This means: settings in Project _tenant produce auto-docs in Project _tenant; settings in Project acme-research produce auto-docs in Project acme-research. No cross-scope bleeding.

Per Scope:

  1. SettingService.findAll(tenant, "project", projectId) returns all settings in the scope. Keys ai.provider.<instance>.{type,apiKey,baseUrl} are grouped per <instance>.
  2. Determine protocol type — same order as in the resolver: ai.provider.<instance>.type, otherwise wireType from _vance/model/<instance>/_provider.yaml, otherwise instance == ProviderType.wireName(). Unknown type → skip + WARN. The order must match §3: an instance that the resolver chats with, but Discovery skips, has a button that does nothing without an error message.
  3. Decrypt API key via SettingService.getDecryptedPassword(...). For providers with requiresApiKey() and an empty key → skip + DEBUG.

  4. Call AiModelService.findProvider(type).listAvailableModels(ProviderListingRequest). Each provider bean implements the same SPI; internally, the respective listing API is called:

    Provider Endpoint Data Fields
    Anthropic GET /v1/models (x-api-key) id only
    OpenAI / OpenAI-wire Gateways GET /v1/models (Bearer) id only
    Gemini GET /v1beta/models?key=... id + inputTokenLimit
    Ollama GET /api/tags (no auth) id (with :-tag convention)
    OllamaCloud GET /api/tags (Bearer) id (with :-tag)
    LM Studio GET /v1/models (no auth needed) id only
  5. Write two documents per found model. a) Auto-Doc to _vance/model-auto/<instance>/<slug>.yaml (Slug encoding: : → - plus wireName: field; / → nested subdirectories): wireName (if necessary) plus the limit observations (contextWindowTokens, maxOutputTokens, ownedBy, each only if present) as well as discoveredBy: discovery-job, discoveredAt: <ISO-8601>. Classifications (kind, capabilities) and Pricing are never in the auto-doc — the auto layer outranks bundled per field and is completely overwritten on each run (see below). OpenAI itself only provides id/owned_by; the parser normalizes the limit dialects: context_window/context_length/context_size/max_context_length/max_input/max_input_tokens/max_tokens → contextWindowTokens; max_output/max_output_tokens/max_completion_tokens → maxOutputTokens. b) Price-Doc (Pricing paragraph below): if the endpoint reports prices (cortecs EUR/MTok, OpenRouter per-token-USD → converted), Discovery processes them — but at the place where prices belong: the manual layer (_vance/model/<instance>/<slug>.yaml), as a machine-owned file with an auto: true marker. The contract (ownership by marker, Bistromath pattern):
  • No file present → Discovery creates it: auto: true + wireName (if necessary) + pricing: block. No manual work.
  • File with marker → remains machine-owned: every Discovery run updates the prices (prices change — no one wants to re-enter them).
  • File without marker → operator-owned: Discovery never touches it again, what’s in it wins via the cascade.
  • Operator takes over a machine-generated file by removing the marker (and then corrects it as desired); returns it to automation by re-creating the marker.

Why this detour instead of prices directly into the auto-doc (§3a.6 Step 5a): the auto layer outranks bundled per field and is completely overwritten per run — a gateway price as an auto value would shadow curated bundled prices (including currency mix bundled-USD vs. gateway-EUR in the usage report) as soon as instance names coincide (e.g., cortecs via the openai instance). The manual layer, on the other hand, is the right place: it is the home of prices, and the marker rule guarantees that an existing operator/bundled entry is not displaced — only a missing file is added or a machine-owned one is updated. What still does not happen: deriving prices from other sources (LightLlm/Web-Search against pricing pages — hallucination risk, deliberately not included in v1). Models whose endpoint does not report prices remain unpriced until an operator writes a manual doc (see Eddie-Manual manuals/ai-model-catalog for the workflow).

kind — also deliberately not auto-discovered. Discovery writes exclusively observations (wire name, limits, owned-by), never classifications (kind, capabilities) — and never prices (see above). The reason is the cascade direction: the auto layer is above the bundled layer, so an asserted kind: chat would overwrite a correct bundled kind: image. Specifically: Gemini lists gemini-2.5-flash-image with generateContent — it fits every “chat-capable” filter, but is an image model. Before this fix, a Discovery run reclassified it as a chat model, causing it to disappear from listAllImages and thus from the Fenchurch alias pickers, while ai.alias.default.image continued to point to it → invalid_choice when saving the LLM form. Providers therefore no longer report kind at all (DiscoveredModelInfo has no such field).

Idempotence: Auto-docs are always overwritable; every Discovery run writes them anew. Operator edits live in the disjoint _vance/model/** path and are thus automatically safe — no discoveredBy check needed.

Refresh after job end: ModelCatalog.refresh() is called internally directly, so that the new auto-docs become visible without a second REST call.

Scheduled Discovery (ModelDiscoveryTick) — opt-in, default off: A Spring tick can run Discovery regularly for all Tenants, 2 minutes after boot and then every 6 hours (vance.ai-models.discovery.interval / vance.ai-models.discovery.initial-delay, ISO-8601 durations). Activated via vance.ai-models.discovery.enabled: true — Default false: a local Dev Brain should not make listing calls against every configured endpoint every few hours; production sets the flag. Master-Pod-guarded — only the lease holder writes (Single-Pod: guard is a no-op); failure isolation per Tenant as in the service itself (a dead endpoint does not stop other Tenants). If active, new models appear automatically and auto: true prices remain current without human intervention; manual triggers remain the immediate way.

Quota Check: All Levels Simultaneously

This is important: a call consumes quota at every level. If Mike works in Project X:

Call: 3000 Tokens
  → Account acc_mike: 142k → 145k / 500k   ✓
  → Project proj_X: 890k → 893k / 5000k    ✓
  → Team team_nlp: 4.2M → 4.203M / 20M     ✓
  → Tenant: no limit                       ✓
  → All OK → Call allowed

If the Project budget is exhausted but Mike’s personal budget is not:

  → Account acc_mike: 142k / 500k          ✓
  → Project proj_X: 4.998M / 5000k         ✗ OVER
  → Blocked: "Project quota exhausted"

4. A New LLM Object Per Call

The Brain does not hold a global ChatClient. For each LLM call, a fresh client is created with the correct credentials:

public class LlmFactory {

    public ChatClient createForSession(Session session, String purpose) {
        Account account = session.getAccount();
        Project project = session.getProject();
        
        // Model based on purpose — Preferences cascade
        String model = resolveModel(purpose, account, project);
        
        // Find Provider and Key — Cascade
        ProviderCredential cred = resolveProvider(model, account, project);
        
        // Check quota — all levels
        quotaService.checkAllLevels(account, project, estimatedTokens);
        
        // Build a fresh ChatClient
        return ChatClient.builder()
            .model(cred.getProvider(), model)
            .apiKey(cred.decryptApiKey())
            .build();
    }
}

4.1 Provider Decorator Chain

Each per-call AiChat (built via StandardAiChat) stacks an optional decorator layer around the langchain4j ChatModel of the provider. This same layer performs two tasks against the raw response — Think-tag stripping and model-specific message parsing — and is hooked in only if at least one of the two is active:

engine / Light-LLM / StandardAiChat.ask
  ↑ cleaned ChatResponse  ←  engines never know which markup was stripped
                          ←  or which inline tool-call format was rewritten
SanitizingChatModel       ←  optional — wrapped iff modelInfo.stripThinkTags = true
                          ←  OR a MessageParser is bound for the model
  ↑ raw ChatResponse
LoggingChatModel          ←  AiTraceLogger debug-log + LlmTraceRecorder persistence
                          ←  Trace sees RAW (forensic audit, training-data analysis)
  ↑ raw ChatResponse
provider ChatModel        ←  AnthropicDirectChatModel / GoogleAiGeminiChatModel /
                          ←  OpenAiChatModel / OllamaChatModel / …

Streaming has the same topology with SanitizingStreamingChatModel as the outermost layer — it is only hooked in if a MessageParser is bound; the parser acts in the onCompleteResponse callback, partial tokens pass through unchanged.

Contract of the layers:

  • LoggingChatModel is always active. Writes every request/response to de.mhus.vance.brain.ai.trace logger plus optionally via LlmTraceWriter to the LlmTraceDocument collection. ALWAYS sees the raw model response including any existing reasoning markup or inline emitted tool calls.
  • SanitizingChatModel processes the response in two stages:
    1. Message Parser Stage (if a MessageParser is bound, see §4.1.1): rewriting of the AiMessage — typically synthesizes ToolExecutionRequests from inline emitted tool-call text (Gemma-4) or repairs malformed function.arguments (DeepSeek-V4).
    2. Think-Tag-Strip Stage (if modelInfo.stripThinkTags() == true): clones the ChatResponse with a cleaned AiMessage (Tool-Execution-Requests remain untouched), so that all consumers above (Engines, chat_messages persistence, History-Replay, Judges) only see the final user text.

    Default behavior (both flags inactive, no parser bound): no wrap, no overhead.

Why not in the Engine: Engines (Arthur, Eddie, Ford, Marvin, …) consume response.aiMessage().text() and aiMessage().toolExecutionRequests() and should NOT have to know per provider/model which markup to expect or which model emits text instead of structured tool calls. Sanitizer and MessageParser are cross-cutting and belong on the provider side — analogous to the trace logger.

Raw-Thoughts-Capture (instead of pure discarding). The raw narration that the model streams during the turn (for reasoning models, the <think>…</think> monologues from Qwen3/DeepSeek-R1/Granite or Harmony-analysis-channels for GPT-OSS) is valuable to the user — they want to be able to reread their live-seen thoughts later. Instead of discarding them, the Structured-Action family of Engines (Arthur, Eddie) accumulate the raw response text of each loop iteration verbatim per turn (analogous to the historyTagSink pattern, in the TurnReasoningBuffer) and write it as a separate thinking field to the ChatMessageDocument during persistence — separate from the final content (which comes from the structured Action message field). Nothing is filtered: the thinking value is exactly the text the client saw streaming. Models that only emit the Action tool call without free text (typically non-reasoning models like Claude) provide empty text → null, no field. The thinking content goes to clients via ChatMessageDto/ChatMessageAppendedData; the Web UI shows it as an expandable, verbatim rendered “Thoughts” area below the response (see web-ui.md §6.5). Important: on the streaming path, the strip stage does not apply (partial tokens pass through raw), so the raw narration for the chat turn is in the aggregated aiMessage().text() and is encapsulated there.

4.1.1 MessageParser SPI

Some LLMs do not reliably provide structured tool calls (Gemma-4 family via LM Studio/llama.cpp serializes them as text with Gemma-internal <|"|> quote token; DeepSeek-V4-Pro appends trailing garbage to valid function.arguments). Instead of burdening every Engine path with fallback logic, there is a small SPI:

package de.mhus.vance.brain.ai.parser;

public interface MessageParser {
    String name();                          // e.g., "gemma4", "deepseek-v4"
    ChatResponse parse(ChatResponse raw);   // pure transform, no I/O
}

Concrete @Component implementations (Gemma4MessageParser, DeepSeekV4MessageParser) live under de.mhus.vance.brain.ai.parser. The MessageParserRegistry (also a Spring Bean) collects them by name(). AbstractChatProvider.createChat resolves the parser from modelInfo.messageParser() and passes it to StandardAiChat.

Contract of each implementation:

  • Pure — no I/O, no Mongo access, no Engine state reads.
  • Defensive — if the response already looks good (e.g., aiMessage().hasToolExecutionRequests() == true), the parser MUST return the input verbatim. The cascade can enable the parser for a model that usually has the quirk; occasional clean turns must pass through unchanged.
  • Stateless — Spring instantiates one bean per parser; many chat calls run concurrently.

Resolution Cascade for messageParser (outermost → innermost; innermost wins):

1. Project-Layer YAML    _vance/model/<provider>/<model>.yaml    (explicit)
2. _tenant-Layer YAML    _vance/model/<provider>/<model>.yaml    (explicit)
3. _vance-Layer YAML     _vance/model/<provider>/<model>.yaml    (explicit)
4. Bundled per-model    vance-defaults/_vance/model/<...>.yaml  (explicit)
5. Bundled Quirks-File   vance-defaults/model-quirks.yaml        (Pattern match by Name)
6. null  (no parser — pass-through)

Layers 1–4 are the normal §3a.3 cascade over the messageParser field. Layer 5 is the Quirks Default Layer — a single bundled YAML with glob patterns that applies across providers:

# vance-brain/src/main/resources/vance-defaults/model-quirks.yaml
rules:
  - match: "deepseek-v4*"
    messageParser: "deepseek-v4"
  - match: "gemma-4*"
    messageParser: "gemma4"

match is a case-insensitive glob (* = 0+ characters, ? = 1 character) against the wire model name. First match wins. Provider-agnostic — the same deepseek-v4-pro via real DeepSeek, OpenRouter, local vLLM gets the same parser.

Why this way:

  • Zero-config for the default case. New models of the Gemma-4 or DeepSeek-V4 family are automatically recognized without having to maintain per-model YAMLs anywhere.
  • One place for “known quirks”. model-quirks.yaml is versioned with the Brain code and visible in code review — no scattered settings.
  • Specific beats general. An override in the per-model YAML (messageParser: null or a different parser name) wins against the pattern.
  • No auto-detection in the hot path. Selection is data-driven and resolved once during model resolution — no per-turn regex race. Defensive canParse checks live within the concrete parser implementation, not in routing.

Adding a new parser:

  1. @Component under de.mhus.vance.brain.ai.parser with MessageParser implementation.
  2. A line in model-quirks.yaml with match pattern + messageParser name (if cross-provider).
  3. Unit test with the concrete LLM output snippet (preferably verbatim from Brain log).

The name() is validated by MessageParserRegistry on boot; unknown names in messageParser fields produce a WARN log line, but no boot error — the response then simply passes through unchanged.

4.1.2 Request Quirks — outputTokenParam, unsupportedParams, reasoningEffortWhenOff

The same quirks file carries not only response rewrites but also request quirks: three ModelInfo fields that describe the dialect a model requires on the wire. All three were verified against openai:gpt-5.6-sol on 2026-08-10 (ModelWireProbeAiTest, see below) — each one was a hard HTTP 400 that killed the turn.

outputTokenParam (max_tokens max_completion_tokens, Default max_tokens) — which field carries the output cap; the OpenAiProvider reads it when building the request builder. OpenAI’s reasoning models (o-series, gpt-5 upwards) reject max_tokens with HTTP 400 and require max_completion_tokens; all other OpenAI-wire endpoints (cortecs, LM Studio, Ollama, GLM, DeepSeek) only know the historical field. This makes it a per-model fact, not a provider switch.

unsupportedParams (list of temperature/top_p/top_k/frequency_penalty/presence_penalty/seed/stop, Default empty) — sampling knobs that the model rejects. AbstractChatProvider nulls them centrally before buildModels, i.e., uniformly for every provider; the model then runs on its own defaults instead of rejecting the request. This is not cosmetic: AiChatOptions sets temperature via builder default, so a reasoning model without this entry is dead in every turn. An empty list in YAML is a statement (“accepts everything”) and beats the pattern; a missing field inherits it.

reasoningEffortWhenOff (String, Default unset) — what is sent when Vance does not want reasoning. Normal case: send nothing at all. Reasoning-native models (gpt-5.x) think by default and then reject this in combination with function tools ("Function tools with reasoning_effort are not supported … set reasoning_effort to 'none'") — since every Engine turn carries a tool manifest, the off state must be explicitly stated. Consequence in the catalog: the gpt-5.6-* models deliberately do not have a thinking capability, otherwise a thinking: medium Recipe could override the pin again.

rules:
  - match: "gpt-5*"
    outputTokenParam: "max_completion_tokens"
    unsupportedParams: ["temperature", "top_p", "frequency_penalty", "presence_penalty", "stop"]
    reasoningEffortWhenOff: "none"

Cascade identical to messageParser (per-model YAML beats pattern beats default), with two additions:

  • Resolution per field. A rule may carry messageParser, outputTokenParam, or both; the first rule that sets the requested field is sought. This prevents a parser rule from obscuring a later token rule for the same model.
  • Patterns also apply without catalog entry. The synthetic ModelInfo fallback for an unknown model also goes through the quirks — an uncataloged gpt-5 derivative would otherwise die on the 400 error just because no one wrote its YAML.

Onboarding a new model — ModelWireProbeAiTest. The dialect facts above were not guessed, but measured. qa/ai-test/.../ModelWireProbeAiTest (opt-in via VANCE_MODEL_PROBE=1) boots a Brain, resolves the default model configured via settings, and fires one single-sentence call per request parameter — without Engine, without Session, without tool manifest. Each parameter is a separate test method, so the Surefire report is the compatibility matrix; the provider error message in the assert is the phrasing that belongs in the model YAML.

VANCE_MODEL_PROBE=1 \
VANCE_INIT_SETTINGS_FILE=$PWD/confidential/init-settings-<model>.yaml \
    ./wb qa ModelWireProbeAiTest

The probe covers the parameter level; the tool level (e.g., the reasoning_effort-with-tools conflict) only shows up in a real Engine turn — for this, a small E2E like EddieLearnAiTest with the same init-settings file is sufficient.

4.2 Provider Implementation Convention — AbstractChatProvider

Each concrete provider (AnthropicProvider, GeminiProvider, OpenAiProvider, OllamaProvider, OllamaCloudProvider, LmStudioProvider) inherits from AbstractChatProvider. The base class handles the recurring orchestration:

final AiChat createChat(config, options):
  1. validate getType().wireName() == config.provider()
  2. modelInfo = modelCatalog.lookupOrDefault(...)
  3. effective = applyOptionGates(options, modelInfo)  ← optional override
  4. parser   = messageParserRegistry.get(modelInfo.messageParser()).orElse(null)
  5. built    = buildModels(config, effective, modelInfo) ← abstract
  6. return new StandardAiChat(... modelInfo.stripThinkTags(), responseSanitizer, parser)
  7. catch RuntimeException → AiChatException wrap

Subclasses only implement buildModels(...) (langchain4j ChatModel/StreamingChatModel construction) and optionally override applyOptionGates(...) for provider-specific option transformations (e.g., Anthropic’s capability/cache kill). Provider-specific @Value configs (BaseURL, cacheEnabled) remain in the concrete class.

What this pattern buys: new cross-cutting layers (e.g., a future rate-limiter decorator, a new trace layer) land in one place (AbstractChatProvider.createChat or StandardAiChat) instead of six times in each provider. Concrete examples:

  • stripThinkTags field + SanitizingChatModel layer was added without any of the six providers getting their own logic for it.
  • messageParser field + MessageParserRegistry lookup + Gemma-4-/DeepSeek-V4-parser (see §4.1.1) were later also added exclusively in the template method — provider constructors only received the registry as an additional parameter.

5. Token Tracking

Every LLM call is tracked — with reference to all scope levels:

usage_record:
  id: usage_001
  account_id: acc_mike
  project_id: proj_5               # for project budget tracking
  team_id: team_nlp                # for team budget tracking
  tenant_id: tenant_acme
  session_id: sess_abc123
  thinkProcessId: tp_12
  node_id: node_7_1
  provider: anthropic
  model: claude-sonnet-4
  purpose: execution
  tokens_input: 2450
  tokens_output: 890
  tokens_total: 3340
  cost_usd: 0.012
  credential_source: project       # where the key came from: account | project | team | tenant | local
  duration_ms: 2340
  timestamp: 2026-04-23T14:30:00

Quota Status Per Level 

quota_status:
  - scope: account/acc_mike
    period: daily
    tokens_used: 142000
    tokens_limit: 500000

  - scope: project/proj_5
    period: monthly
    tokens_used: 890000
    tokens_limit: 5000000

  - scope: team/team_nlp
    period: monthly
    tokens_used: 4200000
    tokens_limit: 20000000

Warning Thresholds (Configurable Per Level)

Threshold Action
80% Notification to affected Accounts
95% Notification + Brain switches to Light model
100% Tasks blocked, only local LLM (if available)

6. Local LLM as Free Tier 

tenant:
  llm_providers:
    - provider: local_ollama
      model: llama-3.3-70b
      endpoint: http://localhost:11434
      api_key: null
      priority: 99
      quota: unlimited
      capabilities:
        planning: adequate
        execution: limited
        light: good

Free Accounts without their own key only use the local LLM. Projects with grant budgets use cloud LLMs. The fallback to local works transparently.

7. Model Routing Per Task Type

Task Type Typical Model Why
Tree Planning Strong (Claude Sonnet) Tree structure requires reasoning
Extract Medium (Gemini Flash) Simpler
Analyze / Verify Strong Requires critical thinking
Synthesize Strong Creative + analytical
Tag / Classify Light (local LLM) Simple categorization
Brain Linker Medium Many calls, budget-friendly

Configurable per Account, Project, and Think Process:

engine:
  id: tp_12
  llm_overrides:
    extract: gemini-2.5-flash
    synthesize: claude-sonnet-4

Preferences cascade for model selection: Think Process override → Project preferences → Account preferences → Tenant default.

7a. Model Characteristics: Context-Discipline vs. Training-Trust

Models of the same performance class behave differently disciplined when tool outputs or user corrections contradict their training snapshot. For Vancetope, this is not academic — the architecture is designed for workers to retrieve fresh data and for the model to synthesize an answer from it. If the model ignores fresh data in favor of its training, the entire research pipeline is ineffective.

Two Dimensions

Dimension Description Failure Mode
Context-Discipline How strongly does the model respect the content of its context window (tool outputs, previous Assistant replies with source attribution, user corrections) against its training? Model rejects fresh facts as “not official” / “does not exist”, even though its own web research confirmed them one message earlier
Tool-Loop-Persistence Can the model sustain a multi-stage tool loop until the answer (Decompose → Call Recipe → Use Result)? Model runs dozens of LLM calls without a single tool call because it “knows” what the answer is

Observed Profiles (as of 2026-06-15)

This table is empirical, not exhaustive. It is based on Vancetope-internal research sessions (see analysis/sess_*/COMPARISON.md for the underlying traces) and is explicitly allowed to be falsified as soon as vendor releases change behavior. Model character drift is real — Anthropic / OpenAI / Google regularly patch tool-use behavior.

Model Context-Discipline Tool-Loop-Persistence Vancetope Recommendation
gemini:gemini-2.5-pro Weak — known training override bias in factual questions Medium Codegen / Boilerplate / Language translation. Not to be set as default analyze/web alias for research workflows.
gemini:gemini-2.5-flash Weak (same bias mechanism as Pro) Strong Cheap tier for classification, tag recognition, Inbox triage — where training knowledge already provides the answer
openai:gemma-4-26b-a4b-it (via cortecs) Strong — explicitly documents its own conflicts between research and training, decides in favor of research Strong Research default, analyze / web / code alias suitable
openai:deepseek-v4-pro Strong (in observed sessions) Strong Frontier research, Deep-Think (Marvin)
anthropic:claude-sonnet-4-6 Strong Strong Frontier research, Deep-Think — preferred if budget available
anthropic:claude-opus-4-7 / 4-8 Strong Very strong Premium research, agentic tool loops
Local Ollama models (qwen3:30b, gemma4:31b-mlx, …) Variable per model — test Variable Free tier; verify suitability per model before setting them to a research alias

Consequence for Recipe Defaults

Vancetope Recipes have a sensible default model alias (default:analyze, default:web, default:deep). Tenant operators configure the binding of these aliases. The recommendation to operators:

  • default:fast may be a model with weak Context-Discipline (classification / triage benefits from training trust)
  • default:analyze, default:web, default:deep should be bound to models with strong Context-Discipline, otherwise Ford workers and Marvin trees are self-serving theater

In Vancetope v1, this is not enforced by an automatic selector — Tenants have their own provider contracts. The character information in this table is decision support for init-settings-*.yaml and Web UI setting forms, not code.

Symptoms in the Trace (Diagnostic Heuristic)

If a Tenant complains about poor research results, check in this order:

  1. Tool-Loop-Persistence — how many LLM calls per tool call? If ratio > 20:1 in a Marvin/Ford session: model is self-serving. Change.
  2. Context-Discipline — Search the final reply for phrases like “does not exist”, “not publicly available”, “no official model”, even though earlier Assistant replies or tool results prove the opposite. If such a passage is found: model is overriding context with training. Change.
  3. Source Attribution — Final reply without [source: url] anchors in research tasks means: model did not take its own tool outputs seriously as evidence. Marginal case — can predict symptom 2.

What this table does NOT do

  • No general “Model A is better than Model B” ranking. Gemini 2.5 Pro is excellent for many tasks (code generation, language, multi-modal); just not for research workflows that need to reconcile fresh facts against training.
  • No promise that profiles remain stable. Vendor updates can reverse behavior within weeks.
  • No default action — Operations decides per Tenant what binds.

8. Multi-Provider Failover 

Provider A (Priority 1) → Error → Retry → Error
  → Provider B (Priority 2) → Success
  → Log: "Failover from anthropic to google"

llm_config:
  failover:
    max_retries: 2
    retry_delay_ms: 2000
    fallback_enabled: true
    fallback_to_local: true

8.1 Empty Completions — Retryable vs. Deterministic

ResilientStreamingChatModel retries not only exceptions, but also successful but empty completions (no text, no tool call) — these come via onCompleteResponse and would never reach the exception path. Two causes, two treatments:

Case Symptom Treatment
Provider glitch / Model collapse empty, finish ≠ LENGTH (e.g., Gemini with STOP) retry, max. 3 attempts with backoff, then chain advance
Output cap exhausted empty, finish = LENGTH no retry — chain advance directly, otherwise deliver empty response

The LENGTH case is deterministic: the model has exhausted its max_tokens budget (for reasoning models, typically entirely in reasoning_content), an identical re-request hits the same wall. Retries only cost tokens and wall-clock there. Chain advance remains allowed — a fallback entry can carry a larger cap and save the turn.

In both cases, the empty response is delivered unchanged to the caller (including finishReason), not converted into an error. The Engine decides what happens next.

Engine side: StreamedReply. Engines no longer reduce a completion directly to its AiMessage — that discarded the finishReason, precisely the information needed to distinguish the two cases. Instead, StreamedReply (vance-brain/.../ai/) encapsulates message + finishReason + maxOutputTokens (the latter read from the request, not from the Model config — the user should see the value that was actually on the wire) and offers isEmpty(), atOutputCap(), and emptyReplyMessage(collapseMessage, stateNote). The truncation text is formulated once there (diagnosis + buttons are the same everywhere); the non-truncation phrasing remains per Engine, because how it is parked and whether a rephrase even helps differs.

Consumers:

  • Frankie — persists the Assistant message and parks BLOCKED.
  • Trillian-Control — has in addition to the resilience layer its own one-time retry; this is skipped for LENGTH and the message says “Output limit” instead of “Rephrase the question”. Exit remains IDLE.
  • Trillian-User — headless, a silent turn is normal and there is no user to notify. Instead of a message, for LENGTH there is a WARN line, so that the config cause does not disappear as a “quiet turn” in the log.

9. LLM-Usage Dashboard

Built (v2): two levels, one write path.

Tracking occurs at the seam, not at the call site. A UsageAccountingChatModel sits as a decorator in the model chain, attached in AbstractChatProvider.createChat — the only place through which every chat is built. It is within ResilientChatModel and below the trace log, thus seeing every attempt: retries, chain fallbacks, aborted streams, and failed attempts, all of which were previously invisible. Previously, callers booked themselves, and those who didn’t book didn’t exist — Jeltz (and thus Magrathea’s agent_task), Agrajag, Eddie’s LLM triage, and the Cortex deep-validate ran completely unaccounted for.

Who pays is a mandatory parameter. createChat(config, options, attribution) takes a CallAttribution (vance-shared, an object instead of six loose fields, unpacked all the way to the ledger write). A caller who does not provide it will cause a compile error. Not to be confused with AiChatOptions.tenantId/projectId — these indicate against which level the model catalog resolves (a tenant-pinned Process deliberately leaves projectId empty there), while billing must still name the Project that consumed the tokens. Two questions, two fields.

Instead of engineName, the dimension is called caller: Think Engines by their name, everything else names itself — _light (single-shot helpers: Discovery, Follow-Up, Title-Gen), _triage, _compaction, _fenchurch (images), _rag (embeddings), _deep-validate. “Engine” hasn’t been accurate for a while. Not source: that word in this tree means the opposite direction (Zarniwoop/Centauri/Jaglan — external data coming in).

Two levels, two lifespans:

  • llm_usage_daily is the billing: a bucket per (tenant, UTC-day, project, caller, recipe, model, currency, kind), incremented live via atomic $inc-upsert (deterministic bucketId = the key itself, hashed; duplicate-key race → one retry). No night job, no lease, no watermark — and “today” is complete, instead of needing permanent special treatment in the report. Default retention: forever. Amounts are integer micro-units (cost*Micros, 1e-6 of the currency), not double. These fields are added via $inc once per attempt and never recalculated — and when the detail lines are gone after 60 days, they are the only remaining proof. IEEE-754 accumulation is lossy and order-dependent; irrelevant for a diagnostic counter, but not for the number on the invoice. LlmUsageService.toMicros/fromMicros are the only converters, conversion only happens on read. The detail lines retain double — nothing sums there. Migration: 2026-08-24_001.
  • llm_usage_records is the diagnosis: one line per attempt with Session/Process/Attempt/Rate-Snapshot. Short-lived (60 days success, 14 days failed attempts) and switchable via setting — which is only possible since the daily balance is the billing.

Retention via the cascade (usage.retentionDays, usage.detailRetentionDays, usage.detailRetentionDaysFailed) plus TTL index, as with the Megadodo feed. Detail is tri-state (>0 days / 0 forever / <0 don’t write at all), the daily balance is bi-state — billing is not turned off by a number.

Coverage is reported, not smoothed. A model without a pricing: block is unknown, not free; the line carries actual tokens and 0 cost, and the bucket counts unpricedCalls so the report can say “the amount covers 94% of calls”. To ensure “no block” unambiguously means “price missing”, locally running models (ollama/lmstudio) declare an explicit zero rate. Failed attempts are counted separately and never included in the amount.

One level below is unmeasuredCalls: a successful call for which the provider did not report any token numbers (some Ollama/LM Studio endpoints). Such calls were initially discarded — “a zero line adds no information”. The discarded information was that the call did happen: the endpoint completely disappeared from the report, and a missing line reads as “nothing ran there”. Now it is booked and marked. The difference from unpricedCalls is the question that is open: there, the rate is missing, here, the tokens are missing.

Read side: GET /brain/{tenant}/usage/{summary,by-project,by-model,by-caller,by-recipe} (Tenant-ADMIN, by-engine remains as an alias), rendered in the Insights tab “Usage & Cost”. All five are projections of the same bucket key, summed over buckets instead of millions of individual lines — therefore, a multi-year window can also be answered. detailHorizon in the DTO says from when drill-down is still available; it is read from the oldest existing detail line, not calculated from the setting (which would lie after every change).

LlmCallTracker maintains the live HUD and Prometheus counters — these need the Process and are not billing.

The following mockup is the target sketch (quotas/warnings are not yet built):

Token Usage — Today (Account: Mike)
  ├── Total: 142k / 500k (28%)
  ├── By Project:
  │     Transformer Review: 89k / 5M monthly (1.8%)
  │     Vancetope Architecture: 53k / no project limit
  ├── By Provider:
  │     Anthropic (project key): 89k
  │     Google (personal key): 53k
  └── Estimated daily cost: $0.48

Token Usage — This Month (Project: Transformer Review)
  ├── Total: 890k / 5M (17.8%)
  ├── By Account:
  │     Mike: 540k
  │     Sarah: 350k
  └── Estimated monthly cost: $3.20

10. Summary 

LLM Credentials + Quotas at every level:
  Account → Project → Team → Tenant → Local

Per Call:
  1. Determine model (Purpose → Preferences cascade)
  2. Find Provider + Key (Credential cascade)
  3. Check quota (ALL levels, all must be OK)
  4. Instantiate fresh ChatClient
  5. Execute call
  6. Track usage at all levels

Every LLM call is a fresh client. Credentials and budgets can be at the Account, Project, Team, or Tenant level. All applicable quotas are checked and charged simultaneously.

11. Implementation Phases

Phase What
v1 One Provider, one Key (Environment), global tracking
v1.5 Account-Level Keys, Provider Resolution, Quota per Account
v2 Project-Level Keys + Quotas, Multi-Provider, Failover
v2 Local LLM as Free Tier, Model Routing per Task Type
v3 Team/Tenant-Level Sharing, Usage Dashboard, Warnings
*See also: identity-credentials architektur-scopes-clients execution-modes-trigger*