Method

When comma publishes a new driving model, as a changed model file on openpilot master or as an upload to commaai/openpilot_driving_models, one fixed suite scores it and the site gets a page for it. Each score is compared with lebowski, the 877M model from July 2026, which this truck runs, and with the model whose weights were first public just before it (in a section that model was not run in, the latest earlier model that was).

The suite has two halves. The public half uses comma's public data and models, and anyone can rerun it from the published code. The truck half uses recorded drives from one 2011 Mercedes G55, which openpilot does not support. Those drives stay private, so only their scores are published. The front page leads with the truck half; the public half is a check of the harness (below).

What each section measures

Each section on a model page lists its own metrics, units, sample size and condition. A section that was not run for a model says so; it is never filled in from another model.

Unit of evidence and intervals

Every value is a mean over units: a 60 s segment in the public sections, a window (one stretch of one recorded drive) in the truck sections. Units are grouped by the drive they come from: comma's 39 moving public segments come from 23 drives, and the 28 closed-loop ones from 19. The interval beside a value is a 95% bootstrap over drives: resample whole drives with replacement, recompute the mean over all their units, repeat 4000 times, and keep the 2.5th and 97.5th percentiles. Three adjacent minutes of one drive therefore count as one draw, not three.

With fewer than 10 drives there is no interval. In simulation, a nominal 95% interval of this kind covers the true mean about 90% of the time at 10 drives and 79% at 4, and less when the data are skewed. The truck sections come from three or four drives, so they report the mean, the median and win counts, including the number of drives in which the model did better, and no interval.

A delta compares a model with a baseline on the units both were scored on. It is model minus baseline, computed per unit and then bootstrapped the same way, so differences between cars, roads and light that affect both models cancel. The fraction beside it counts the units where the model did better; for lateral action rate and the lead of the lateral request, which have no better direction, it counts the units where the model's value was lower.

A blue dot or a red diamond means the delta's interval excludes zero (a grey triangle, for a measure with no better direction); a ring means the data do not separate the two models; a dash means there is no interval. Each page shows dozens of these comparisons with no correction for multiple tests, so a delta whose interval only just excludes zero is weak evidence. Where one segment dominates a mean, the median beside it shows it.

Two measures need care. The lead of the lateral request is the shift, searched from 0 to 1.5 s, at which the model's lateral request best correlates with the car's later lateral motion: a larger value means the request matches motion further ahead, so it comes earlier. Smoothing a request also moves that peak, so the lead is not a timing measure on its own and has no better direction. A value at either end of the search is only a bound and is left out, so its count can be below the section's. The plan speed error at 3 s reads the model's plan output, which does not separate models built on one frozen base: it is identical for the four October models on base f22e06e5.

Every public interval can be recomputed from the per-segment CSVs: uv run python -m geval.reproduce in eval/. A ranking ranks models by the median distance from the recorded path at 2 s in closed loop, with a 95% interval of the median that resamples drives. Two models are separated when the interval of the difference of their medians, on the units both have, excludes zero; a model's rank is 1 plus the number of models separated from it with a lower median. The front page ranks by the G55 closed loop once it has 10 or more drives; the window values stay private, so the ranking is published as rankings/truck_closed_loop.json (medians, intervals, counts, ranks). Until then the front page orders models by the G55 windows closer to the recorded path than lebowski at 2 s, with no rank. The harness check ranks comma's public segments the same way; uv run python -m geval.ranking prints it from the published CSVs.

Why open loop and closed loop can disagree

Open loop asks whether the model's output matches what a human did on a recorded drive. The output never changes the next frame, so a model can copy the human closely and still drift once its own steering moves the car. Closed loop feeds the steering back through a world model, and small errors build up.

Measured in the 2026-10-27 deep dive, the two disagree. comma's recent models are a frozen base policy with a residual head trained by reinforcement learning on top. With the residual switched off, the base had the lowest open-loop lateral error of any model on comma's public segments (0.074 m/s², against lebowski's 0.109). In the first closed-loop run on nine truck windows (before the current eighteen), after a small yaw disturbance, the same base ended about 1.8 m from the driven path at 4 s. lebowski ended 0.4 m away, and a policy that does not steer at all 2.3 m; the same models with their residual ended 1.2 to 1.3 m closer to the path than their base alone, in 9 of 9 windows. On comma's public segments the residual lowers the lateral action rate by 33 to 40% and raises the lead of the lateral request by 0.17 to 0.32 s. Aligned directly with its own base, the residual's lateral request comes 0.13 to 0.26 s earlier on 16 of comma's cars (0.10 to 0.15 s on the G55); the rest of the difference goes with the smoother request. Open-loop scoring counts both as error.

This matches published work: on the nuPlan benchmark, open-loop and closed-loop planning scores were negatively correlated (Dauner et al., 2023). The site reports both kinds of score side by side and does not combine them.

What the truck sections can and cannot show

They show how each model behaves with a camera height, a vehicle and a steering response that comma's fleet does not cover, and whether a ranking on comma's cars holds on a vehicle unlike them.

• They come from one truck. Nothing in them says how a model drives another vehicle.
• There are few windows. Each section shows its count, and the conditions (day, night, speed, camera setup) are not balanced.
• Others cannot reproduce them: the drives are private, and only aggregate scores and a few checked stills are published.
• Closed loop is a simulation. The world model is weakest at night, rolls out 7.2 s at most, and pulls late frames toward what was actually recorded, so differences that appear only past about 4 s carry little weight. In a separate test it showed 0 to 20% of an added lateral offset 3 to 4 s after it, so the front-page ranking uses 2 s.
• No newer model has steered the truck. It drives on lebowski, and every other model is scored by replay.

Stills are a few camera frames per condition with the model's path drawn on them, each checked so that nothing in it identifies the truck, its owner or the place.

Downloads

One results file per model, in the format described in SCHEMA.md, and one CSV per public section with a row per segment. G55 sections publish scores only.