Vance Benchmark — model × test

Generated 2026-08-05T18:20:08.735929Z — 1 models, 60 tests. Each cell shows the LATEST result; click for the run's .md report.

model \ test	anti-hallucination					document-kinds					how-do-i-reflex					inline-kinds				learn-action				mermaid-variety									script-javascript			script-python		tool-family										voice-style
model \ test	rejectsCalendarCreateEvent	rejectsDiagramTool	rejectsDocSave	rejectsListAdd	rejectsRecordsCreate	createsApplicationKind	createsChartKind	createsDiagramKind	createsGraphKind	createsMindmapKind	discoversAmbiguousMetaphor	discoversComposedUnknown	discoversInventedFeature	discoversJargonRequest	discoversUnknownTerm	rendersChartInline	rendersDiagramInline	rendersGraphInline	rendersMindmapInline	learnsFactAppend	learnsFactReplace	learnsPersonaAppend	learnsPersonaReplace	emitsC4ContextDiagram	emitsErDiagram	emitsGanttDiagram	emitsGitGraph	emitsJourneyDiagram	emitsPieDiagram	emitsSequenceDiagram	emitsStateDiagram	emitsTimelineDiagram	executesJavaScriptJsonTransform	executesJavaScriptPrimes	executesJavaScriptSum	executesPythonPrimes	executesPythonStringReverse	picksCalendarFamily	picksDocFamily	picksGraphFamily	picksHookFamily	picksListFamily	picksRecordsFamily	picksSchedulerFamily	picksScratchFamily	picksSheetFamily	picksTreeFamily	voiceAcronymExpansion	voiceFenceForCodePath	voiceFenceForLongList	voiceFenceForTable	voiceFenceNotMisused	voiceModeOffMidConversation	voiceNumbersSpeakable	voiceQuestionEndsOpenly	voiceShortBulletsAllowedInline	voiceShortProseReply	voiceSpokenPartNoMarkdownLeak	voiceSttToleranceCutWord	voiceSttToleranceHomophone
openai-deepseek-v4-pro	OK 1.00 14.1s 103.2k → 537 brain · foot	OK 1.00 24.7s 105.0k → 1.8k brain · foot	OK 1.00 9.4s 50.4k → 292 brain · foot	OK 0.60 5.0s 48.6k → 184 brain · foot	OK 1.00 20.7s 187.7k → 717 brain · foot	OK 1.00 11.1s 105.4k → 551 brain · foot	OK 1.00 9.3s 77.2k → 536 brain · foot	OK 1.00 11.1s 130.4k → 545 brain · foot	OK 1.00 9.5s 77.9k → 497 brain · foot	OK 1.00 7.5s 77.3k → 325 brain · foot	OK 0.70 4.7s 49.0k → 236 brain · foot	OK 1.00 55.1s 152.2k → 720 brain · foot	OK 1.00 13.4s 100.2k → 790 brain · foot	OK 1.00 16.0s 152.1k → 591 brain · foot	OK 1.00 7.2s 73.7k → 403 brain · foot	OK 1.00 12.1s 50.5k → 243 brain · foot	OK 1.00 6.8s 50.4k → 186 brain · foot	OK 1.00 83.4s 50.9k → 255 brain · foot	OK 1.00 4.7s 50.6k → 207 brain · foot	OK 1.00 3.2s 24.6k → 161 brain · foot	OK 1.00 7.0s 24.5k → 122 brain · foot	OK 1.00 3.6s 24.7k → 164 brain · foot	OK 1.00 3.8s 24.7k → 185 brain · foot	OK 1.00 10.4s 104.6k → 801 brain · foot	OK 1.00 8.2s 103.9k → 635 brain · foot	OK 1.00 11.6s 104.2k → 1.0k brain · foot	OK 1.00 11.0s 104.1k → 932 brain · foot	OK 1.00 17.3s 139.1k → 1.3k brain · foot	OK 1.00 15.3s 130.6k → 650 brain · foot	OK 1.00 27.2s 138.9k → 1.5k brain · foot	OK 1.00 26.7s 103.7k → 547 brain · foot	OK 1.00 25.9s 189.9k → 895 brain · foot	OK 1.00 9.5s 49.0k → 265 brain · foot	OK 1.00 11.2s 49.1k → 379 brain · foot	OK 1.00 9.9s 49.0k → 171 brain · foot	OK 1.00 9.0s 49.3k → 392 brain · foot	OK 1.00 9.4s 73.5k → 261 brain · foot	OK 1.00 7.9s 78.8k → 464 brain · foot	OK 1.00 3.4s 48.9k → 163 brain · foot	OK 1.00 8.6s 98.2k → 390 brain · foot	OK 1.00 44.7s 125.4k → 894 brain · foot	OK 1.00 9.2s 73.5k → 257 brain · foot	OK 1.00 101.1s 48.8k → 198 brain · foot	OK 1.00 37.2s 26.3k → 301 brain · foot	OK 1.00 44.4s 48.8k → 150 brain · foot	OK 1.00 6.5s 73.5k → 330 brain · foot	OK 1.00 7.0s 73.4k → 255 brain · foot	OK 1.00 4.2s 24.8k → 222 brain · foot	FAIL 0.00 9.2s 49.9k → 209 brain · foot	FAIL 0.00 - brain · foot	FAIL 0.00 110.3s 78.8k → 2.2k brain · foot	OK 1.00 2.0s 24.8k → 68 brain · foot	OK 1.00 68.1s 74.5k → 936 brain · foot	OK 1.00 46.7s 49.7k → 156 brain · foot	OK 1.00 26.3s 52.2k → 458 brain · foot	OK 1.00 4.8s 24.8k → 203 brain · foot	FAIL 0.00 67.0s 50.2k → 473 brain · foot	OK 1.00 58.7s 162.3k → 986 brain · foot	OK 1.00 58.8s 49.9k → 206 brain · foot	OK 1.00 13.6s 52.1k → 732 brain · foot

Per-model summary

Overall = all tests of the latest run. chat = subset relevant for chat / audio2audio agents (arthur, eddie) — action-choice, anti-hallucination, discovery, learning, inline rendering, script-execution. worker = subset relevant for worker agents (ford, marvin, vogon, …) — production tasks, kind generation, tool-family targeting. Some capabilities (anti-hallucination, tool-family, scripts) count for both.

model	overall				chat		worker		duration
model	pass	fail	missing	rate	pass	rate	pass	rate	duration
openai-deepseek-v4-pro	56	4	0	93%	42/46	91%	34/34	100%	25:48

Aggregated matrix (all runs)

Pass-rate per (model, test) over all runs of that combination. Cell shading: green ≥ 80%, yellow ≥ 50%, red < 50%. Outliers trimmed (Winsorized) for the mean-score when n ≥ 5.


	anti-hallucination					document-kinds					how-do-i-reflex					inline-kinds				learn-action				mermaid-variety									script-javascript			script-python		tool-family										voice-style
	rejectsCalendarCreateEvent	rejectsDiagramTool	rejectsDocSave	rejectsListAdd	rejectsRecordsCreate	createsApplicationKind	createsChartKind	createsDiagramKind	createsGraphKind	createsMindmapKind	discoversAmbiguousMetaphor	discoversComposedUnknown	discoversInventedFeature	discoversJargonRequest	discoversUnknownTerm	rendersChartInline	rendersDiagramInline	rendersGraphInline	rendersMindmapInline	learnsFactAppend	learnsFactReplace	learnsPersonaAppend	learnsPersonaReplace	emitsC4ContextDiagram	emitsErDiagram	emitsGanttDiagram	emitsGitGraph	emitsJourneyDiagram	emitsPieDiagram	emitsSequenceDiagram	emitsStateDiagram	emitsTimelineDiagram	executesJavaScriptJsonTransform	executesJavaScriptPrimes	executesJavaScriptSum	executesPythonPrimes	executesPythonStringReverse	picksCalendarFamily	picksDocFamily	picksGraphFamily	picksHookFamily	picksListFamily	picksRecordsFamily	picksSchedulerFamily	picksScratchFamily	picksSheetFamily	picksTreeFamily	voiceAcronymExpansion	voiceFenceForCodePath	voiceFenceForLongList	voiceFenceForTable	voiceFenceNotMisused	voiceModeOffMidConversation	voiceNumbersSpeakable	voiceQuestionEndsOpenly	voiceShortBulletsAllowedInline	voiceShortProseReply	voiceSpokenPartNoMarkdownLeak	voiceSttToleranceCutWord	voiceSttToleranceHomophone
openai-deepseek-v4-pro	1/1 μ=1.00	1/1 μ=1.00	1/1 μ=1.00	1/1 μ=0.60	1/1 μ=1.00	2/2 μ=1.00 σ=0.00	2/2 μ=1.00 σ=0.00	2/2 μ=1.00 σ=0.00	2/2 μ=1.00 σ=0.00	1/2 μ=0.50 σ=0.71	1/2 μ=0.35 σ=0.49	2/2 μ=1.00 σ=0.00	2/2 μ=1.00 σ=0.00	1/2 μ=0.50 σ=0.71	2/2 μ=1.00 σ=0.00	1/1 μ=1.00	1/1 μ=1.00	1/1 μ=1.00	1/1 μ=1.00	1/1 μ=1.00	1/1 μ=1.00	1/1 μ=1.00	1/1 μ=1.00	2/2 μ=1.00 σ=0.00	2/2 μ=1.00 σ=0.00	2/2 μ=1.00 σ=0.00	2/2 μ=1.00 σ=0.00	2/2 μ=1.00 σ=0.00	1/2 μ=0.50 σ=0.71	2/2 μ=1.00 σ=0.00	2/2 μ=1.00 σ=0.00	2/2 μ=1.00 σ=0.00	2/2 μ=1.00 σ=0.00	2/2 μ=1.00 σ=0.00	2/2 μ=1.00 σ=0.00	1/2 μ=0.50 σ=0.71	2/2 μ=1.00 σ=0.00	2/2 μ=1.00 σ=0.00	2/2 μ=1.00 σ=0.00	2/2 μ=1.00 σ=0.00	2/2 μ=1.00 σ=0.00	2/2 μ=1.00 σ=0.00	2/2 μ=1.00 σ=0.00	2/2 μ=1.00 σ=0.00	2/2 μ=1.00 σ=0.00	2/2 μ=1.00 σ=0.00	1/2 μ=0.50 σ=0.71	1/1 μ=1.00	0/1 μ=0.00	0/1 μ=0.00	0/1 μ=0.00	1/1 μ=1.00	1/1 μ=1.00	1/1 μ=1.00	1/1 μ=1.00	1/1 μ=1.00	0/1 μ=0.00	1/1 μ=1.00	1/1 μ=1.00	1/1 μ=1.00

Model stability (across runs)

Per-model rollup of the aggregated matrix. samples is total (test, run) data-points. overall = pass-rate across all samples. chat / worker = pass-rate restricted to the capability subset relevant for that agent kind (mapping documented above the Per-model summary). score μ / σ use the trimmed effectiveScore across all samples — higher μ + lower σ = better + more stable.

model	tests	samples	overall		chat		worker		μ	σ
model	tests	samples	rate	pass / n	rate	pass / n	rate	pass / n	μ	σ
openai-deepseek-v4-pro	60	94	89%	84/94	88%	58/66	94%	59/63	0.89	0.31

Run history

openai-deepseek-v4-pro — 14 runs (latest 2026-08-05T18:17:17.911387Z)

started	class	pass	fail	duration	report
2026-08-05T18:17:17.911387Z	MermaidVarietyBenchmark	9	0	2:50	md
2026-08-05T18:13:43.342355Z	HowDoIReflexBenchmark	5	0	2:38	md
2026-08-05T17:31:11.749349Z	DocumentKindsBenchmark	5	0	1:33	md
2026-08-05T17:26:07.567046Z	ToolFamilyBenchmark	10	0	4:03	md
2026-08-05T17:24:00.690083Z	ScriptCapabilityBenchmark	5	0	46.3s	md
2026-08-05T14:36:13.206588Z	MermaidVarietyBenchmark	8	1	4:57	md
2026-08-05T14:34:54.763823Z	LearnActionBenchmark	4	0	13.9s	md
2026-08-05T14:30:26.612877Z	HowDoIReflexBenchmark	3	2	3:33	md
2026-08-05T14:25:56.930620Z	DocumentKindsBenchmark	4	1	3:17	md
2026-08-05T14:19:56.143249Z	ToolFamilyBenchmark	9	1	5:01	md
2026-08-05T14:18:10.630659Z	ScriptCapabilityBenchmark	4	1	36.5s	md
2026-08-05T14:15:02.848784Z	AntiHallucinationBenchmark	5	0	1:42	md
2026-08-05T14:12:54.589436Z	InlineKindsBenchmark	4	0	56.2s	md
2026-08-05T13:59:34.273323Z	VoiceStyleBenchmark	9	4	11:03	md