openpilot driving model evaluations
Every comma.ai openpilot driving model, including Hugging Face uploads, scored with one fixed suite on drives from a 2011 Mercedes G55, a car openpilot does not support, in closed loop and open loop. The same suite runs on comma's public drives as a check of the harness.
- G55 drives
- 18 closed-loop windows from 4 drives; ranks and intervals from 10 drives
- Models
- 8 tracked, 3 scored on the G55 in closed loop
- Last scored
- 2026-10-27 21:27 UTC
- Newest
6b5b705f, first public 2026-10-17, scored 2026-10-27- New models
- checked every 20 minutes since 2026-10-27 on Hugging Face and openpilot's branches; no new model since. Results published by hand.
G55 closed loop (primary): windows closer to the recorded path than lebowski at 2 s
18 windows from 4 drives. Each model drives a model of the G55's steering and powertrain in worldmodel-4B from recorded G55 windows. Intervals and ranks need 10 or more drives; until then models are ordered by this count, with no rank. Under each name: comma's public segments with comma's ideal car, the harness check.
8d7366832026-10-10, shipped in openpilot v0.11.2, on master from 2026-10-10comma's segments: 5.6 cm at 2 s
1365126a2026-10-14, merged to openpilot master 2026-10-14, again 2026-10-24comma's segments: 7.4 cm at 2 s
comma's segments: 7.2 cm at 2 s
6b5b705f2026-10-17, Hugging Face upload only, not shipped96244b1f2026-10-15, Hugging Face upload only, not shipped870a48232026-10-08, Hugging Face upload only, not shippedf78ed37d2026-09-18, merged to openpilot master 2026-09-181a4211752026-09-11, merged to openpilot master 2026-09-11The tick in each bar is half the windows. Medians in cm are distance from the recorded path at 2 s.
Why 2 s: in a separate test on 18 G55 windows from 4 drives, worldmodel-4B followed an added yaw command through 2 s, but 3 to 4 s after an added yaw or lateral command it showed 0 to 20% of the 1.2 to 2.1 m of extra offset. After 2 s a lateral error can shrink in the simulation without the driving model correcting it.
Numbers behind the chart
| Model | G55 closed loop, closer than lebowski | G55, 2° heading error, closer than lebowski | G55, 0.5 m offset, closer than lebowski | comma's segments, median at 2 s (cm) |
|---|---|---|---|---|
| Mountain Dew (8d736683) | 13/18 windows, 4/4 drives; median 15.8 | 10/18 windows, 2/4 drives; median 34.1 | 10/18 windows, 4/4 drives; median 28.5 | 5.6 |
| MD v3 / Flip Flop (1365126a) | 12/18 windows, 4/4 drives; median 16.2 | 8/18 windows, 3/4 drives; median 38.0 | 7/18 windows, 2/4 drives; median 29.5 | 7.4 |
| lebowski | reference; median 22.7 | reference; median 39.7 | reference; median 29.2 | 7.2 |
| 6b5b705f | pending | pending | pending | pending |
| N16 nav model (96244b1f) | pending | pending | pending | pending |
| 870a4823 | pending | pending | pending | pending |
| Cinque v3 (f78ed37d) | pending | pending | pending | pending |
| Cinque v2 (1a421175) | pending | pending | pending | pending |
Over time
Each model at the date its weights were first public, joined in that order. A new model is added when it is scored.
Mountain DewMD v3 / Flip Flopother comma modelson openpilot master or shippedHugging Face upload onlylebowski, July 2026 (win panels: half the windows)not scored yetbars: 95% interval
G55 closed loop: windows closer to the recorded path than lebowski at 2 s
G55, 2° heading error: windows closer than lebowski at 2 s
G55, 0.5 m offset: windows closer than lebowski at 2 s
- 16–23 cm
Median distance at 2 s on the G55 (18 windows), against 6–7 cm on comma's segments (28 segments).
Detail
Closed loop, median distance from the recorded path at 2 s: 16 to 23 cm on the G55 with the model of its steering and powertrain (3 models; 18 windows from 4 drives), and 6 to 7 cm on comma's public segments with comma's ideal car (3 models; 28 segments from 19 drives).
- −29% to −38%
Lateral action rate of each model after lebowski, against lebowski's, open loop (39 segments).
Detail
Open loop on comma's public segments (39 segments from 23 drives): lateral action rate (RMS) of each model scored after lebowski is 29 to 38% lower than lebowski's. Their lateral gain, the size of the lateral request against the car's motion (1 = same size), is 0.70 to 0.82; lebowski's is 0.83. Lead of the lateral request over the car's motion, relative to lebowski: Mountain Dew +0.23 s (35 segments), MD v3 / Flip Flop +0.09 s (34 segments); a larger lead means the request best matches the car's motion further ahead. In a separate test aligning each with its own base (residual off) on 16 of comma's cars, their lateral request came 0.26 s (Mountain Dew) and 0.13 s (MD v3 / Flip Flop) earlier; the rest of the difference goes with the smoother request.
- 0.62–1.78 m
Distance from the recorded path at 2 s on chevy_silverado_s1, the 3 models scored in closed loop; every other segment within 0.35 m.
Detail
Closed loop on comma's public segments, distance from the recorded path at 2 s: on one segment, chevy_silverado_s1, the 3 models scored are 0.62 to 1.78 m off. On every other segment, every model is within 0.35 m. The means over segments are 0.09 to 0.15 m and the medians 0.06 to 0.07 m.
Log
Model releases and scoring runs from the results files; suite changes from CHANGELOG.md. Newest first.
- suiteThe G55 sections lead: the ranking is G55 closed loop at 2 s, ranked with intervals once it has 10 or more drives and until then ordered by windows closer than lebowski. comma's public drives move to a harness check near the bottom; the world-model player moves to the method page.
- suiteThe closed-loop video is replaced by a player: the world model's 256×128 frames as generated, the paths drawn by the page over them, one view at a time.
- suiteOver-time charts and this log replace the one-off results section.
- suiteClosed-loop video of one public segment, with each model's path drawn on the generated view. fe1a5ed
- suite"Lateral lag" corrected to the lead of the lateral request over the car's motion: a larger value means the request comes earlier. No better direction is marked for it or for lateral action rate. 8cebf52
- suiteModels ranked by the median distance from the recorded path at 2 s in closed loop; the 4 s "primary" horizon is dropped. 8cebf52
- suiteResults recomputed with the new intervals from the stored per-segment values, no new runs. 1b06fee
- suiteIntervals resample whole drives; no interval below 10 drives, so G55 sections report win counts. Lead values at the 0 and 1.5 s search limits are left out. 102710f
- suiteRelease watcher started: Hugging Face commaai/openpilot_driving_models and openpilot's branches, every 20 minutes. 7856299
- suiteG55 recovery after a 0.5 m offset added, 18 windows. a98409f
- suiteClosed loop on comma's public segments added: 28 segments in worldmodel-4B, comma's ideal car, seeds 0 and 1. f0ec83f
- suiteFirst edition: 8 models, scores backfilled from the 2026-10-27 deep dive. 3a0056f
20 earlier entries
- scoredlebowski scored, suite geval-1: comma's public drives, open loop; comma's public drives, closed loop; G55, open loop; G55, closed loop; G55, recovery tests
- scoredCinque v2 (1a421175) scored, suite geval-1: comma's public drives, open loop; G55, open loop
- scoredCinque v3 (f78ed37d) scored, suite geval-1: comma's public drives, open loop; G55, open loop
- scored870a4823 scored, suite geval-1: comma's public drives, open loop; G55, open loop
- scoredMountain Dew (8d736683) scored, suite geval-1: comma's public drives, open loop; comma's public drives, closed loop; G55, open loop; G55, closed loop; G55, recovery tests
- scoredMD v3 / Flip Flop (1365126a) scored, suite geval-1: comma's public drives, open loop; comma's public drives, closed loop; G55, open loop; G55, closed loop; G55, recovery tests
- scoredN16 nav model (96244b1f) scored, suite geval-1: comma's public drives, open loop; G55, open loop
- scored6b5b705f scored, suite geval-1: comma's public drives, open loop; G55, open loop
- modelMD v3 / Flip Flop (1365126a) merged to openpilot master again
- modelMountain Dew (8d736683) shipped in openpilot v0.11.2
- model6b5b705f uploaded to Hugging Face
- modelN16 nav model (96244b1f) uploaded to Hugging Face
- modelMD v3 / Flip Flop (1365126a) merged to openpilot master
- modelMountain Dew (8d736683) merged to openpilot master
- model870a4823 uploaded to Hugging Face
- modelCinque v2 (1a421175) uploaded to Hugging Face
- modelCinque v3 (f78ed37d) merged to openpilot master
- modelCinque v3 (f78ed37d) uploaded to Hugging Face
- modelCinque v2 (1a421175) merged to openpilot master
- modellebowski merged to openpilot master
The models
Newest first, by the date the weights were first public. Each links to its full scores.
6b5b705f2026-10-17 Hugging Face upload only, not shipped
Lateral action rate 30% lower, lateral error 26% higher and longitudinal error 19% higher than lebowski (39 segments). Against N16 nav model: lower lateral error; higher longitudinal error.
N16 nav model
96244b1f2026-10-15 Hugging Face upload only, not shipped
Lateral action rate 29% lower and lateral error 40% higher than lebowski (39 segments). Against MD v3 / Flip Flop: higher lateral lead, lateral error and longitudinal error. Navigation inputs fed zeros.
MD v3 / Flip Flop
1365126a2026-10-14 Merged to openpilot master 2026-10-14, again 2026-10-24
Lateral action rate 32% lower and lateral error 18% higher than lebowski (39 segments). Against Mountain Dew: lower lateral action rate, lateral lead, lateral error and longitudinal error.
Mountain Dew
8d7366832026-10-10 Shipped in openpilot v0.11.2, on master from 2026-10-10
Lateral action rate 29% lower and lateral error 32% higher than lebowski (39 segments). Against 870a4823: higher lateral action rate, lateral lead and lateral error.
870a48232026-10-08 Hugging Face upload only, not shipped
Lateral action rate 34% lower and lateral error 23% higher than lebowski (39 segments). No metric separates from Cinque v3.
Cinque v3
f78ed37d2026-09-18 Merged to openpilot master 2026-09-18
Lateral action rate 36% lower and lateral error 26% higher than lebowski (39 segments). Against Cinque v2: lower lateral error and longitudinal error; higher lateral action rate.
Cinque v2
1a4211752026-09-11 Merged to openpilot master 2026-09-11
Lateral action rate 38% lower and lateral error 36% higher than lebowski (39 segments).
lebowski
2026-07-01 Merged to openpilot master 2026-07-01
Reference model; the G55 drives with it. Highest lateral action rate and shortest lateral lead of the models scored.
How it works
Open loop
The model runs on recorded camera frames. Its lateral and longitudinal requests are compared with the car's measured motion that followed.
Closed loop
The model's requests move a simulated car, and comma's public world model, worldmodel-4B, generates the next camera frame from the new position, 7.2 s per episode. Scored as distance from the recorded path.
The G55
A 2011 Mercedes G55 (W463) with recirculating-ball steering and the camera 1.45 m above the road. openpilot does not support it, so its drives are not in comma's training data. Its closed loop uses a model of the G55's steering and powertrain in place of comma's ideal car.
comma publishes its own test reports in model_reports; the newest there is Cinque v3, from 2026-09-18. Nearest equivalents:
| Here | Nearest test in comma's model_reports |
|---|---|
| Lateral action rate, open loop | jitter |
| Distance from path, closed loop, comma's ideal car | MLSim lateral (analyse_lat_mlsim), their world model as the simulator |
| G55 recovery: 0.5 m lateral offset or 2° heading error | no direct match; closest is hugging, recovery under a lateral bias |
| Braking lead car (not built yet) | unintended lead following |
The full method, with a closed-loop example from worldmodel-4B.
Harness check on comma's public drives
Purpose: to check that the harness reproduces sensible behaviour on comma's own data, and to let others rerun it. The same suite runs on comma's public segments, open loop and closed loop with comma's ideal car; per-segment results and code are published, and python -m geval.reproduce recomputes every interval from them.
Closed loop on comma's public segments: median distance from the recorded path at 2 s
28 segments from 19 drives, comma's ideal car in worldmodel-4B; 95% interval resampling drives. Rank as in the G55 chart.
Lateral action rate, open loop, comma's segments (m/s³)
Lead of the lateral request over the car's motion, open loop, comma's segments (s)
For researchers
- Every number is a mean over units: a 60 s segment of a public drive, or a window of a G55 drive. Units are grouped by the drive they come from, and the 95% intervals resample whole drives. Below 10 drives there is no interval, so the G55 sections report means and win counts only.
- A paired difference compares two models on the units both were scored on; the count beside it is the units where the model did better.
- Per-segment CSVs, one results file per model (schema) and the code are public;
python -m geval.reproducerecomputes every public interval from the CSVs. Downloads - The G55 drives stay private; their scores are published, not the frames.
better, 95% interval excludes zero Better, 95% interval excludes zeroworse, 95% interval excludes zero Worse, interval excludes zerolower, 95% interval excludes zero; no better directionhigher, 95% interval excludes zero; no better direction Lower or higher, no better direction (action rate, lead)not separated, interval includes zero Not separatedno interval (too few drives) No interval: fewer than 10 drives
| Model | G55 distance from path at 2 sG55, closed loop | G55 at 2 s after a 0.5 m offsetG55, recovery tests | Lateral action ratecomma's public drives, open loop | Lateral leadcomma's public drives, open loop | Lateral errorcomma's public drives, open loop | Distance from path at 2 s, closed loopcomma's public drives, closed loop |
|---|---|---|---|---|---|---|
6b5b705f | not run | not run | 0.464[0.371, 0.571]lower, 95% interval excludes zero; no better direction−0.200 vs lebowski | 0.558[0.446, 0.680]higher, 95% interval excludes zero; no better direction+0.169 vs lebowski | 0.137[0.108, 0.168]worse, 95% interval excludes zero+0.028 vs lebowski | not run |
N16 nav model 96244b1f | not run | not run | 0.471[0.380, 0.576]lower, 95% interval excludes zero; no better direction−0.193 vs lebowski | 0.539[0.429, 0.660]higher, 95% interval excludes zero; no better direction+0.154 vs lebowski | 0.152[0.119, 0.185]worse, 95% interval excludes zero+0.043 vs lebowski | not run |
MD v3 / Flip Flop 1365126a | 0.211no intervalno interval (too few drives)−0.057 vs lebowski | 0.341no intervalno interval (too few drives)−0.008 vs lebowski | 0.451[0.353, 0.566]lower, 95% interval excludes zero; no better direction−0.212 vs lebowski | 0.470[0.377, 0.574]higher, 95% interval excludes zero; no better direction+0.088 vs lebowski | 0.129[0.101, 0.159]worse, 95% interval excludes zero+0.020 vs lebowski | 0.087[0.056, 0.135]better, 95% interval excludes zero−0.060 vs lebowski |
Mountain Dew 8d736683 | 0.229no intervalno interval (too few drives)−0.039 vs lebowski | 0.338no intervalno interval (too few drives)−0.011 vs lebowski | 0.474[0.374, 0.595]lower, 95% interval excludes zero; no better direction−0.190 vs lebowski | 0.615[0.504, 0.736]higher, 95% interval excludes zero; no better direction+0.226 vs lebowski | 0.143[0.113, 0.176]worse, 95% interval excludes zero+0.035 vs lebowski | 0.096[0.048, 0.182]better, 95% interval excludes zero−0.051 vs lebowski |
870a4823 | not run | not run | 0.438[0.345, 0.546]lower, 95% interval excludes zero; no better direction−0.226 vs lebowski | 0.514[0.414, 0.626]higher, 95% interval excludes zero; no better direction+0.123 vs lebowski | 0.134[0.106, 0.163]worse, 95% interval excludes zero+0.025 vs lebowski | not run |
Cinque v3 f78ed37d | not run | not run | 0.422[0.343, 0.511]lower, 95% interval excludes zero; no better direction−0.242 vs lebowski | 0.466[0.366, 0.577]higher, 95% interval excludes zero; no better direction+0.065 vs lebowski | 0.137[0.108, 0.166]worse, 95% interval excludes zero+0.028 vs lebowski | not run |
Cinque v2 1a421175 | not run | not run | 0.412[0.336, 0.498]lower, 95% interval excludes zero; no better direction−0.252 vs lebowski | 0.492[0.383, 0.603]higher, 95% interval excludes zero; no better direction+0.092 vs lebowski | 0.147[0.113, 0.183]worse, 95% interval excludes zero+0.039 vs lebowski | not run |
| lebowski | 0.268no interval | 0.349no interval | 0.664[0.541, 0.792] | 0.373[0.316, 0.444] | 0.109[0.085, 0.134] | 0.148[0.069, 0.281] |
comma models not scored here
- Be Right Here (big), 2026-08-02, 877M class: Reverted on 2026-08-14 ("Revert big RL model").
- BIG RL
23e6a04e, 2026-08-31, ~383M: The first smaller model; PR #38681, branch remove-avgpool. - ResAction
7a933c1c, date unknown, ~383M: Published only as a pickle.