Vance Benchmark

Generated 2026-09-13T22:54:25.343809Z — 7 models, 79 tests, 553 samples.

Category score (all runs, weighted)

Per cell: mean score of that capability's tests, each test first averaged over its own runs — so a capability with 13 tests does not outweigh one with 2. total is the weighted mean over the categories that have data (factor in the column header); cov is how many of the categories this model actually ran — rows measured on less than half of them are greyed, because a high total over two categories is not a result. Sub-line in a cell is tests×samples.

Latest run per test

Newest result for each (model, test); the cell is shaded by its score, so partial credit is visible. Click for the run's .md report.

Per-model summary (latest run per test)

Same three-level maths as the category table, but over the latest result per test instead of all runs — this is the snapshot, the table at the top is the trend. chat = subset relevant for chat / audio2audio agents (arthur, eddie); worker = subset relevant for worker agents (ford, marvin, vogon, …). Some capabilities count for both.

Aggregated matrix (all runs)

Mean score per (model, test) over all runs of that combination — this is the test value the category table is built from. Sub-line is n runs and, from two runs on, the standard deviation: a high σ means the model is inconsistent rather than weak.

Model stability (across runs)

Sample-level rollup, kept next to the category table on purpose: score here is the flat mean over every (test, run) data-point, so comparing it with the weighted total above shows how much of a model's number comes from the capabilities it was measured on most often. pass-rate is the binary verdict for reference. Score μ is Winsorized (10% top + bottom trimmed) from five samples on; σ is over the raw samples.

Run history