openpilot driving model evaluations
Every comma.ai openpilot driving model, including Hugging Face uploads, scored with one fixed suite on comma's public drives and on drives from a 2011 Mercedes G55, in open loop and in closed loop.
Closed loop: each model drives in comma's world model, worldmodel-4B, for 7.2 s from a recorded start. comma's public segments with comma's ideal car: 28 segments from 19 drives. G55 with a model of its steering and powertrain: 18 windows from 4 drives, too few drives for an interval.
| Rank, model, first public, status | comma's public segmentsmedian distance from the recorded path at 2 s, 95% interval | G55closer to the recorded path than lebowski at 2 s | G55, 2° heading errorcloser than lebowski at 2 s | G55, 0.5 m offsetcloser than lebowski at 2 s |
|---|---|---|---|---|
1sharedMD v3 / Flip Flop 1365126a2026-10-14 Merged to openpilot master 2026-10-14, again 2026-10-24 | 7.4 cm[3.9, 8.6] | 12 of 18 windows4 of 4 drives; median 16.2 cm | 8 of 18 windows3 of 4 drives; median 38.0 cm | 7 of 18 windows2 of 4 drives; median 29.5 cm |
1sharedMountain Dew 8d7366832026-10-10 Shipped in openpilot v0.11.2, on master from 2026-10-10 | 5.6 cm[3.4, 6.4] | 13 of 18 windows4 of 4 drives; median 15.8 cm | 10 of 18 windows2 of 4 drives; median 34.1 cm | 10 of 18 windows4 of 4 drives; median 28.5 cm |
| 1sharedlebowski2026-07-01 Merged to openpilot master 2026-07-01 | 7.2 cm[5.5, 9.8] | referencemedian 22.7 cm | referencemedian 39.7 cm | referencemedian 29.2 cm |
–6b5b705f2026-10-17 Hugging Face upload only, not shipped | closed loop pending | |||
–N16 nav model 96244b1f2026-10-15 Hugging Face upload only, not shipped | closed loop pending | |||
–870a48232026-10-08 Hugging Face upload only, not shipped | closed loop pending | |||
–Cinque v3 f78ed37d2026-09-18 Merged to openpilot master 2026-09-18 | closed loop pending | |||
–Cinque v2 1a4211752026-09-11 Merged to openpilot master 2026-09-11 | closed loop pending | |||
- Rank is 1 plus the number of models whose median is lower with a paired 95% interval, resampling drives, that excludes zero. Models with the same rank are not separated. G55 drives: those where the model is closer on the mean over the drive's windows.
- Why 2 s: in a separate test on 18 G55 windows from 4 drives, the frames
worldmodel-4Bgenerated turned with an added yaw command through 2 s. 3 to 4 s after an added yaw or lateral command, with 1.2 to 2.1 m of extra lateral offset executed, lebowski read 0 to 20% of that offset from the lane in the generated frames. After 2 s, a lateral error can shrink in the simulation without the driving model correcting it. - Median over segments, because one segment moves the mean (see Results). Distances at 4 s and 7.2 s are on each model's page.
Results
Measured by this suite, as of 2026-10-27. The reference is lebowski (877M parameters, July 2026), the model the G55 drives with.
- −29% to −38%
Open loop on comma's public segments (39 segments from 23 drives): lateral action rate (RMS) of each model scored after lebowski is 29 to 38% lower than lebowski's. Their lateral gain, the size of the lateral request against the car's motion (1 = same size), is 0.70 to 0.82; lebowski's is 0.83. Lead of the lateral request over the car's motion, relative to lebowski: Mountain Dew +0.23 s (35 segments), MD v3 / Flip Flop +0.09 s (34 segments); a larger lead means the request best matches the car's motion further ahead. In a separate test aligning each with its own base (residual off) on 16 of comma's cars, their lateral request came 0.26 s (Mountain Dew) and 0.13 s (MD v3 / Flip Flop) earlier; the rest of the difference goes with the smoother request.
Lateral action rate against the lead of the lateral request, comma's public segments, open loop
lebowskiMountain DewMD v3 / Flip FlopOther comma modelsOne dot per model; means over comma's public segments, open loop (39 with movement; the lead excludes values at the 0 and 1.5 s search limits). Lead: how far ahead of the car's measured lateral motion the model's lateral request best correlates with it; larger means earlier. Lateral action rate: RMS rate of change of that request. Neither is a driving score: a lower action rate can come from a smaller request as well as a smoother one, and smoothing a request also moves its best-correlating lead. Numbers behind the chart
Model Lateral lead (s) Lateral action rate (m/s³) 6b5b705f 0.56 0.46 N16 nav model (96244b1f) 0.54 0.47 MD v3 / Flip Flop (1365126a) 0.47 0.45 Mountain Dew (8d736683) 0.62 0.47 870a4823 0.51 0.44 Cinque v3 (f78ed37d) 0.47 0.42 Cinque v2 (1a421175) 0.49 0.41 lebowski 0.37 0.66 - 0.62–1.78 m
Closed loop on comma's public segments, distance from the recorded path at 2 s: on one segment, chevy_silverado_s1, the 3 models scored are 0.62 to 1.78 m off. On every other segment, every model is within 0.35 m. The means over segments are 0.09 to 0.15 m and the medians 0.06 to 0.07 m.
- 16–23 cm
Closed loop, median distance from the recorded path at 2 s: 16 to 23 cm on the G55 with the model of its steering and powertrain (3 models; 18 windows from 4 drives), and 6 to 7 cm on comma's public segments with comma's ideal car (3 models; 28 segments from 19 drives).
The models
Newest first, by the date the weights were first public. Each links to its full scores.
6b5b705f2026-10-17 Hugging Face upload only, not shipped
Lateral action rate 30% lower, lateral error 26% higher and longitudinal error 19% higher than lebowski (39 segments). Against N16 nav model: lower lateral error; higher longitudinal error.
N16 nav model
96244b1f2026-10-15 Hugging Face upload only, not shipped
Lateral action rate 29% lower and lateral error 40% higher than lebowski (39 segments). Against MD v3 / Flip Flop: higher lateral lead, lateral error and longitudinal error. Navigation inputs fed zeros.
MD v3 / Flip Flop
1365126a2026-10-14 Merged to openpilot master 2026-10-14, again 2026-10-24
Lateral action rate 32% lower and lateral error 18% higher than lebowski (39 segments). Against Mountain Dew: lower lateral action rate, lateral lead, lateral error and longitudinal error.
Mountain Dew
8d7366832026-10-10 Shipped in openpilot v0.11.2, on master from 2026-10-10
Lateral action rate 29% lower and lateral error 32% higher than lebowski (39 segments). Against 870a4823: higher lateral action rate, lateral lead and lateral error.
870a48232026-10-08 Hugging Face upload only, not shipped
Lateral action rate 34% lower and lateral error 23% higher than lebowski (39 segments). No metric separates from Cinque v3.
Cinque v3
f78ed37d2026-09-18 Merged to openpilot master 2026-09-18
Lateral action rate 36% lower and lateral error 26% higher than lebowski (39 segments). Against Cinque v2: lower lateral error and longitudinal error; higher lateral action rate.
Cinque v2
1a4211752026-09-11 Merged to openpilot master 2026-09-11
Lateral action rate 38% lower and lateral error 36% higher than lebowski (39 segments).
lebowski
2026-07-01 Merged to openpilot master 2026-07-01
Reference model; the G55 drives with it. Highest lateral action rate and shortest lateral lead of the models scored.
How it works
Open loop
The model runs on recorded camera frames. Its lateral and longitudinal requests are compared with the car's measured motion that followed.
Closed loop
The model's requests move a simulated car, and comma's public world model, worldmodel-4B, generates the next camera frame from the new position, 7.2 s per episode. Scored as distance from the recorded path.
The G55
A 2011 Mercedes G55 (W463) with recirculating-ball steering and the camera 1.45 m above the road. openpilot does not support it, so its drives are not in comma's training data. Its closed loop uses a model of the G55's steering and powertrain in place of comma's ideal car.
comma publishes its own test reports in model_reports; the newest there is Cinque v3, from 2026-09-18. Nearest equivalents:
| Here | Nearest test in comma's model_reports |
|---|---|
| Lateral action rate, open loop | jitter |
| Distance from path, closed loop, comma's ideal car | MLSim lateral (analyse_lat_mlsim), their world model as the simulator |
| G55 recovery: 0.5 m lateral offset or 2° heading error | no direct match; closest is hugging, recovery under a lateral bias |
| Braking lead car (not built yet) | unintended lead following |
For researchers
- Every number is a mean over units: a 60 s segment of a public drive, or a window of a G55 drive. Units are grouped by the drive they come from, and the 95% intervals resample whole drives. Below 10 drives there is no interval, so the G55 sections report means and win counts only.
- A paired difference compares two models on the units both were scored on; the count beside it is the units where the model did better.
- Per-segment CSVs, one results file per model (schema) and the code are public;
python -m geval.reproducerecomputes every public interval from the CSVs. Downloads - The G55 drives stay private; their scores are published, not the frames.
better, 95% interval excludes zero Better, 95% interval excludes zeroworse, 95% interval excludes zero Worse, interval excludes zerolower, 95% interval excludes zero; no better directionhigher, 95% interval excludes zero; no better direction Lower or higher, no better direction (action rate, lead)not separated, interval includes zero Not separatedno interval (too few drives) No interval: fewer than 10 drives
| Model | Lateral action ratecomma's public drives, open loop | Lateral leadcomma's public drives, open loop | Lateral errorcomma's public drives, open loop | Distance from path at 2 s, closed loopcomma's public drives, closed loop | G55 distance from path at 2 sG55, closed loop | G55 at 2 s after a 0.5 m offsetG55, recovery tests |
|---|---|---|---|---|---|---|
6b5b705f | 0.464[0.371, 0.571]lower, 95% interval excludes zero; no better direction−0.200 vs lebowski | 0.558[0.446, 0.680]higher, 95% interval excludes zero; no better direction+0.169 vs lebowski | 0.137[0.108, 0.168]worse, 95% interval excludes zero+0.028 vs lebowski | not run | not run | not run |
N16 nav model 96244b1f | 0.471[0.380, 0.576]lower, 95% interval excludes zero; no better direction−0.193 vs lebowski | 0.539[0.429, 0.660]higher, 95% interval excludes zero; no better direction+0.154 vs lebowski | 0.152[0.119, 0.185]worse, 95% interval excludes zero+0.043 vs lebowski | not run | not run | not run |
MD v3 / Flip Flop 1365126a | 0.451[0.353, 0.566]lower, 95% interval excludes zero; no better direction−0.212 vs lebowski | 0.470[0.377, 0.574]higher, 95% interval excludes zero; no better direction+0.088 vs lebowski | 0.129[0.101, 0.159]worse, 95% interval excludes zero+0.020 vs lebowski | 0.087[0.056, 0.135]better, 95% interval excludes zero−0.060 vs lebowski | 0.211no intervalno interval (too few drives)−0.057 vs lebowski | 0.341no intervalno interval (too few drives)−0.008 vs lebowski |
Mountain Dew 8d736683 | 0.474[0.374, 0.595]lower, 95% interval excludes zero; no better direction−0.190 vs lebowski | 0.615[0.504, 0.736]higher, 95% interval excludes zero; no better direction+0.226 vs lebowski | 0.143[0.113, 0.176]worse, 95% interval excludes zero+0.035 vs lebowski | 0.096[0.048, 0.182]better, 95% interval excludes zero−0.051 vs lebowski | 0.229no intervalno interval (too few drives)−0.039 vs lebowski | 0.338no intervalno interval (too few drives)−0.011 vs lebowski |
870a4823 | 0.438[0.345, 0.546]lower, 95% interval excludes zero; no better direction−0.226 vs lebowski | 0.514[0.414, 0.626]higher, 95% interval excludes zero; no better direction+0.123 vs lebowski | 0.134[0.106, 0.163]worse, 95% interval excludes zero+0.025 vs lebowski | not run | not run | not run |
Cinque v3 f78ed37d | 0.422[0.343, 0.511]lower, 95% interval excludes zero; no better direction−0.242 vs lebowski | 0.466[0.366, 0.577]higher, 95% interval excludes zero; no better direction+0.065 vs lebowski | 0.137[0.108, 0.166]worse, 95% interval excludes zero+0.028 vs lebowski | not run | not run | not run |
Cinque v2 1a421175 | 0.412[0.336, 0.498]lower, 95% interval excludes zero; no better direction−0.252 vs lebowski | 0.492[0.383, 0.603]higher, 95% interval excludes zero; no better direction+0.092 vs lebowski | 0.147[0.113, 0.183]worse, 95% interval excludes zero+0.039 vs lebowski | not run | not run | not run |
| lebowski | 0.664[0.541, 0.792] | 0.373[0.316, 0.444] | 0.109[0.085, 0.134] | 0.148[0.069, 0.281] | 0.268no interval | 0.349no interval |
comma models not scored here
- Be Right Here (big), 2026-08-02, 877M class: Reverted on 2026-08-14 ("Revert big RL model").
- BIG RL
23e6a04e, 2026-08-31, ~383M: The first smaller model; PR #38681, branch remove-avgpool. - ResAction
7a933c1c, date unknown, ~383M: Published only as a pickle.