citiesbench v0 · run 2026-09-07 · results never overwritten

Can an AI tell you where to play, truthfully?

We drew the category narrow enough to be measured: answering where-and-whether-to-play questions about real courts, with sourced facts and honest refusals. Not “best sports AI” — that is unwinnable and we would deserve to lose it.

The 240 questions are generated from the index itself, so the right answer is computed rather than judged. Two of the eight question families are refusal families on purpose. A system that answers everything is the failure mode being measured against.

The table

Read our own 240/240 as close to tautological. Our arm reads the same index the gold was computed from, so it proves the retrieval is correct — not that we are clever. The information in this table is in the other three columns. And the index is not yet ground-truthed: 20 named Dallas–Fort Worth venues are being confirmed by phone, and until that is published this number stays on this page and does not appear in any marketing.

By question family

What it says

1. The record removes hallucination, not the weights. Give the same model our rows and invented facilities go 5 → 0, missed refusals 9 → 0. This is the second time we have measured it; the tennis bench found the same thing on 2026-09-04.
2. The model cannot compute. Even holding our rows it scores courts_at 14/14 and type_of 12/12 — then count_within 2/8 and lit_until 1/11. Counting venues inside a radius and calculating a sunset are arithmetic. So the division of labour is measured, not asserted: retrieval and arithmetic to code, language to the model.
3. Honesty rules alone make a model worse at this. lab_bare (rules, no data) scores below lab_naked (no rules): 35/80 against 43/80, with 22 over-refusals. Told to say only what it can support, and given nothing to support it with, it refuses almost everything — safe and useless. Rules without a record are not a product.

A fault in the first scoring, recorded because it changed a headline

The first run scored lab_naked 0/80, all unparsed: it answers in prose with citations and ignores a JSON format instruction. That number was a fact about our harness, not about the model, and published as-is it would have been a lie in our favour. The scorer now reads any arm’s answer the way a person would. We keep this paragraph here because a bench that only publishes the runs that flattered it is not a bench.

Re-run it

The items are seeded, so the same seed gives the same exam to every arm. Results files are never overwritten — a re-run is a new dated file.

make_items.py citiesbench/items_v0.jsonl [n_per_family]
run_arm.py <arm> <items> <out> [limit]
score.py results/*.jsonl

The exam code is not public yet. When it is published it will be published whole — the item generator, the four arms and the scorer — because a benchmark you cannot re-run is a press release. Ask and we will send you the run files for this table today.

Not yet run

SchedulingBench is specified and not built: does a proposed slot satisfy every constraint a player actually stated — their window, their radius, a wet court, a venue whose lights we do not hold? Gold by construction again, so a slot either satisfies the constraints or it does not. It is deliberately scheduled before the scheduling product’s interface: if the solver does not beat the model on computed gold, there is nothing worth building an interface for.

Nothing else is claimed here. Where a bench has not been run, this page says so rather than describing what it would show.