Methodology

DocBench asks a simple question: can this model actually produce the deck or the Word document you would need at work? Every model gets the same tasks, the same tools and the same environment. Two complementary signals come out:

• Benchmark metrics: objective measurements from the runs themselves: success rate, cost, speed, tokens, document size (the Benchmarks page).
• Arena ELO: document quality judged by humans in blind pairwise votes (the Vote and Leaderboard pages). Metrics tell you what a run cost; only people can tell you which document is better.

The current task set (v0.1.0) holds 8 tasks: 4 slide decks (pptx) and 4 Word documents (docx). Each task is an instruction plus a large real-world context document. For example: “create a PowerPoint deck presenting NVIDIA’s financial results” on top of a multi-page annual-report summary, an EU AI Act impact analysis for a bank, a healthcare-systems comparison, an IT resume, or a French sales script. Tasks are versioned; results always record the task version they ran against.

Models do not emit files directly. They drive mini-swe-agent, a deliberately minimal agent that executes bash commands in an isolated Docker container until it submits its work. It is the same harness behind DeepSWE by DataCurve: like them, we benchmark models by giving every one of them the same tiny agent and measuring what it actually produces. The container ships LibreOffice, pandoc, the Node document libraries the skills build with (pptxgenjs, docx-js) and metric-correct fonts for faithful rendering.

Document know-how comes from Agent Skills: instruction packs (a SKILL.mdplus helper scripts) for building pptx and docx files, taken from Anthropic’s skills repository. Skills are copied fresh into the container at the start of every run and pinned to their exact git revision (name@git-sha, flagged -dirty if locally modified), so a leaderboard entry is really a combo of model + reasoning effort + skill set; the same model can be benchmarked with different skills and compared against itself.

Models are accessed through OpenRouter, pinned to high-throughput providers where available, with an explicit reasoning-effort setting per combo (low / medium / high). Every task runs in its own container with a 15-minute wall-time limit, the same for every model: a slow model that would eventually produce a great document still fails if it cannot do so in time. Time spent waiting on provider rate limits is exempt from that clock (throttling is the provider’s condition, not the model’s slowness) and is recorded separately. Runs are checkpointed, so adding a model never re-runs or alters existing results.

A task counts as a success when the agent submits a finished document. Everything else is a failure and counts against the model’s success rate; the failure types are detailed below.

Every task ends with exactly one exit status; only a submitted document counts as a success. Failures keep their cause, and the Task outcomes chart on the Benchmarks page lists them per model in its tooltip:

• Timeout (TimeExceeded): the agent ran past the 15-minute per-task wall-clock limit before submitting.
• Limits exceeded (LimitsExceeded): the agent hit its configured step or cost budget first.
• Repeated malformed responses (RepeatedFormatError): several consecutive responses could not be parsed into a valid tool call (unknown tool, malformed arguments, empty or truncated output). Often transient for a given model.
• Content filter (ContentFilterError): the provider’s safety filter blocked the responses repeatedly, so the agent could never act. Deterministic: the same task hits the same filter on rerun.
• Provider errors (OpenRouterAPIError and friends): the API kept failing after the retry budget (upstream timeouts, 5xx). Usually transient.

Failures count against the success rate and stay visible. Checkpointing skips only successes, so re-running a combo retries its failed tasks: transient provider errors can clear on a rerun, deterministic failures keep failing. A failed task may leave a partial document behind, but it is not counted as a success and never enters the arena.

• Cost: as billed by the provider (for free-routed requests, the upstream inference cost), so it reflects real pricing, not token arithmetic.
• Duration and steps: wall-clock time and number of agent turns per document.
• Tokens: peak context (largest input of any turn) and final output size.
• Document size: slides counted from the pptx file itself, pages from the LibreOffice-rendered docx, which also yields cost per slide and cost per page: the fairest price-per-output comparison across models that write longer or shorter documents.

Averages are computed over successful tasks only.

Every submitted document is rendered page-by-page to images with LibreOffice (using the fonts the documents were authored against). The Vote page then deals blind pairwise matchups: two documents produced for the same task, side by side, model identities hidden by default (an admin can temporarily display them, e.g. for demos), A/B sides randomized. Voters pick the better document or call a tie.

The universe of matchups is every unordered pair of documents produced for the same task. Sampling over pairs (rather than picking a task, then two documents) gives each comparison the same probability regardless of how many documents a task has.

• Full coverage before repeats: matchups are dealt from a shuffled deck of all pairs; every pair comes up exactly once before any pair is shown a second time.
• Least-played pair first, per document type: within the deck, the next matchup is the pair with the fewest votes so far within the matchup’s own task type(slide decks and Word documents each feed their own rating board, so each needs homogeneous coverage), then the fewest votes overall, then the two models’ total game counts (so freshly added models still catch up quickly). Full ties keep the shuffle’s random order.
• Side randomization: a coin flip decides which document appears as A and which as B, cancelling any left/right position bias.
• Info-gain mode (admin toggle): within equally covered pairs, similarly rated models are dealt first, since a matchup with a near 50/50 expected outcome carries the most information per vote.
• Rank bias (admin toggle, composable with the above): sampling can favor pairs near the top or the bottom of the board, to sharpen the ranks that matter most at a given moment.

Administrators can temporarily direct a configurable share of matchups toward a specific model, pair or task type (typically to catch newly added models up to the reliability gate faster). Ratings are always computed from the votes themselves, never from how matchups were sampled.

Ratings come from the Bradley-Terry model — the same method the LMSys / Chatbot Arena leaderboard uses. Rather than nudging a rating up or down after every vote (an online ELO update, which depends on the order votes arrive and lets a handful of votes swing a model), we fit all votes at once by maximum likelihood: we find the set of ratings whose predicted win probabilities best explain every recorded matchup. The predicted probability that side A beats side B is the familiar logistic curve

with a tie counting as half a win and half a loss. A 400-point gap still means 10:1 predicted odds, so the numbers read on the same scale as before, centred at 1000. A mild prior pulls models with very few votes toward the middle (a real vote has to outweigh a small number of virtual draws), which keeps sparse and brand-new models from spiking. Because the fit uses every vote as equal evidence and ignores their order, one vote is only about one part in several thousand: a settled model no longer lurches on two or three votes, and re-running the fit always gives the same board.

The switch from online ELO to Bradley-Terry re-computed the existing ratings; no votes were discarded. If anything, older votes now count for more, since online ELO had let a model’s recent games dominate its number.

There are three leaderboards: Combined, Slides (pptx) and Documents (docx). Each is fit from only the votes on that document type, so a model strong at decks but weak at Word documents shows up as exactly that. The Score-vs-cost charts on the Benchmarks page pair each metric with its matching board: cost per slide against the slides rating, cost per page against the documents rating.

The 95% intervals on the leaderboard come from a bootstrap: the vote history is resampled with replacement 100 times, the Bradley-Terry fit is re-run on each resample, and the interval spans the 2.5th to 97.5th percentile of the resulting ratings. Wide whiskers mean a rating that could still move a lot — typically a model with few votes; overlapping whiskers mean the ranking between two combos is not yet settled.

To know when the leaderboard can be trusted, the Bradley-Terry board is re-fit at a series of cumulative checkpoints through the vote history and three diagnostics are tracked over time:

• Rank stability: Kendall’s τ between the ranking at a checkpoint and the one before it. Values near 1 mean the order has stopped changing.
• Rank movement: the average number of places a model shifts between checkpoints. It shrinks toward 0 as the board settles.
• Pair coverage entropy: the normalized Shannon entropy of the games-per-pair distribution, where 1 means votes are spread perfectly evenly over every pair that existed at that point in the history. Rising entropy means the sampler is filling coverage gaps rather than hammering the same matchups.

A converged board shows rank stability flat near 1, rank movement flat near 0 and entropy climbing toward 1.

A combo with only a handful of votes can sit anywhere on the board by luck. Once a leaderboard has matured (average of at least 8 games per combo), combos with fewer than 40% of the board’s average vote count are hidden until they catch up. The same gate applies to the ELO scatters, evaluated within each board’s own vote counts.

One model is currently excluded from the results and the arena: Qwen3.6 27B (medium reasoning). It completed only 1 of its 8 tasks; the other 7 failed on provider API errors and timeouts rather than on the work itself, and its single surviving document is a one-page resume. Every per-document average would therefore be computed from a single data point, and a rating built on one document says more about that document than about the model. Until the failed tasks are re-run, it appears in no charts or tables, receives no matchups and holds no leaderboard rank.

The votes it collected before the exclusion were kept in the rating replay: they were real human judgments, and removing them would retroactively shift the ratings of the models it faced. If a re-run gives it a full set of results, it returns to the benchmark and the arena with its history intact.

The open-weights flag used by the filters is sourced from OpenRouter’s model metadata (whether a model has published weights on Hugging Face), with a manual override file for the cases the metadata gets wrong. It is a statement about weight availability, not about training data or license terms.

• Every combo faces the identical task set and container image; the skill set is part of the combo and recorded with it.
• Tasks and skills are versioned; a combo’s results record both.
• Failed tasks stay on the board: success rate is part of the result, not a filter.
• Voting is blind and side-randomized; ratings come only from votes, never from the benchmark metrics.