870a4823
2026-10-08 Hugging Face upload only, not shipped
Lateral action rate 34% lower and lateral error 23% higher than lebowski (39 segments). No metric separates from Cinque v3.
Key numbers
The tick is lebowski. Lateral action rate and lateral lead have no better direction; on the others lower is better. Compared with lebowski and with Cinque v3 f78ed37d, the model first public before it.
All scores
better, 95% interval excludes zero Better, 95% interval excludes zeroworse, 95% interval excludes zero Worse, interval excludes zerolower, 95% interval excludes zero; no better directionhigher, 95% interval excludes zero; no better direction Lower or higher, no better direction (action rate, lead)not separated, interval includes zero Not separatedno interval (too few drives) No interval: fewer than 10 drives
comma's public drives, open loop
39 segments from 23 drives. Per-segment CSV.
Measured. comma's public two-camera segments (openpilot CI data), native intrinsics and the device's own calibration, both cameras; every model on the same decoded frames. Truth is what the recording car did next: carState speed and acceleration, yaw rate from the device's cameraOdometry. Segments where the car never moves have no data. 39 of the 45 listed segments have a score, from 23 drives and 20 cars.
| Metric | Mean | vs lebowski | vs Cinque v3 |
|---|---|---|---|
| Lateral errorm/s², lower is better | 0.134[0.106, 0.163]median 0.119 | worse, 95% interval excludes zero+0.025[+0.016, +0.033]better on 8/39 | not separated, interval includes zero−0.003[−0.007, +0.002]better on 21/39 |
| Lateral error at the model's own best leadm/s², lower is better | 0.126[0.099, 0.154]median 0.111 | worse, 95% interval excludes zero+0.025[+0.014, +0.035]better on 9/39 | not separated, interval includes zero−0.002[−0.007, +0.003]better on 19/39 |
| Lateral correlation with the car's later motionhigher is better | 0.841[0.762, 0.912]median 0.945 | not separated, interval includes zero−0.017[−0.049, +0.011]better on 8/39 | not separated, interval includes zero+0.014[−0.004, +0.036]better on 25/39 |
| Lateral gain (1 = same size as the car's motion)closer to 1 is better; the delta is the change in distance from 1 | 0.761[0.684, 0.844]median 0.765 | worse, 95% interval excludes zero+0.061[+0.004, +0.117]better on 11/39 | not separated, interval includes zero−0.004[−0.036, +0.031]better on 20/39 |
| Lateral action rate, RMS (smoothness)m/s³, no better direction | 0.438[0.345, 0.546]median 0.391 | lower, 95% interval excludes zero; no better direction−0.226[−0.272, −0.176]lower on 37/39 | not separated, interval includes zero+0.016[−0.013, +0.052]lower on 23/39 |
| Lead of the lateral request over the car's motions, no better direction | 0.514[0.414, 0.626]median 0.400n = 36 | higher, 95% interval excludes zero; no better direction+0.123[+0.045, +0.219]lower on 10/35 | not separated, interval includes zero+0.006[−0.058, +0.082]lower on 11/34 |
| Longitudinal errorm/s², lower is better | 0.309[0.269, 0.355]median 0.286 | not separated, interval includes zero+0.001[−0.014, +0.019]better on 18/39 | not separated, interval includes zero+0.007[−0.001, +0.017]better on 16/39 |
| Longitudinal correlationhigher is better | 0.732[0.634, 0.818]median 0.847 | not separated, interval includes zero+0.014[−0.013, +0.039]better on 22/39 | not separated, interval includes zero+0.003[−0.016, +0.022]better on 22/39 |
| Longitudinal error while brakingm/s², lower is better | 0.412[0.350, 0.483]median 0.386n = 30 | not separated, interval includes zero−0.023[−0.044, +0.002]better on 15/30 | not separated, interval includes zero+0.004[−0.019, +0.030]better on 17/30 |
| Longitudinal action rate, RMS (smoothness)m/s³, no better direction | 0.894[0.779, 1.028]median 0.776 | lower, 95% interval excludes zero; no better direction−0.439[−0.572, −0.326]lower on 36/39 | not separated, interval includes zero+0.013[−0.048, +0.086]lower on 21/39 |
| Plan speed error at 3 sm/s, lower is better | 0.760[0.657, 0.888]median 0.712 | worse, 95% interval excludes zero+0.044[+0.009, +0.077]better on 12/39 | not separated, interval includes zero+0.002[−0.036, +0.033]better on 13/39 |
comma's public drives, closed loop
Not run for this model yet.
G55, open loop
4 windows from 3 drives.
Measured. The G55's recorded drives replayed as the truck runs the model: narrow camera only, road-plane warp to 1.45 m (plane:1.45). Truth is the truck's CAN speed and yaw rate. Window = unit. 4 windows: 3 on an earlier camera mount, 1 on the current mount; 2 at night.
| Metric | Mean | vs lebowski | vs Cinque v3 |
|---|---|---|---|
| Lateral errorm/s², lower is better | 0.142no intervalmedian 0.154 | no interval (too few drives)+0.014no intervalbetter on 0/40/3 drives | no interval (too few drives)−0.006no intervalbetter on 4/43/3 drives |
| Lateral error at the model's own best leadm/s², lower is better | 0.134no intervalmedian 0.141 | no interval (too few drives)+0.007no intervalbetter on 2/41/3 drives | no interval (too few drives)−0.008no intervalbetter on 4/43/3 drives |
| Lateral correlation with the car's later motionhigher is better | 0.943no intervalmedian 0.946 | no interval (too few drives)−0.018no intervalbetter on 1/41/3 drives | no interval (too few drives)−0.003no intervalbetter on 1/41/3 drives |
| Lateral gain (1 = same size as the car's motion)closer to 1 is better; the delta is the change in distance from 1 | 0.841no intervalmedian 0.837 | no interval (too few drives)+0.101no intervalbetter on 0/40/3 drives | no interval (too few drives)+0.006no intervalbetter on 2/41/3 drives |
| Lateral action rate, RMS (smoothness)m/s³, no better direction | 0.526no intervalmedian 0.542 | no interval (too few drives)−0.430no intervallower on 4/43/3 drives | no interval (too few drives)+0.014no intervallower on 2/42/3 drives |
| Lead of the lateral request over the car's motions, no better direction | 0.463no intervalmedian 0.525 | no interval (too few drives)+0.125no intervallower on 1/41/3 drives | no interval (too few drives)+0.037no intervallower on 0/40/3 drives |
| Longitudinal errorm/s², lower is better | 0.327no intervalmedian 0.331 | no interval (too few drives)+0.013no intervalbetter on 0/40/3 drives | no interval (too few drives)+0.010no intervalbetter on 2/42/3 drives |
| Longitudinal correlationhigher is better | 0.543no intervalmedian 0.573 | no interval (too few drives)−0.039no intervalbetter on 1/41/3 drives | no interval (too few drives)−0.020no intervalbetter on 3/42/3 drives |
| Longitudinal error while brakingm/s², lower is better | 0.536no intervalmedian 0.552 | no interval (too few drives)+0.021no intervalbetter on 1/41/3 drives | no interval (too few drives)−0.048no intervalbetter on 4/43/3 drives |
| Longitudinal action rate, RMS (smoothness)m/s³, no better direction | 1.69no intervalmedian 1.22 | no interval (too few drives)−0.06no intervallower on 3/42/3 drives | no interval (too few drives)+0.42no intervallower on 2/41/3 drives |
| Plan speed error at 3 sm/s, lower is better | 1.30no intervalmedian 1.03 | no interval (too few drives)+0.28no intervalbetter on 1/41/3 drives | no interval (too few drives)+0.26no intervalbetter on 2/42/3 drives |
G55, closed loop
Not run for this model yet.
G55, recovery tests
Not run for this model yet.
Limits
- Open-loop scores compare a model's action with what the recording driver or car did; a model that drives differently but well is scored as wrong. comma's RL residual makes the lateral action smoother, and earlier than its base's by 0.10 to 0.26 s (measured on the G55 and on 16 of comma's cars); both score as error here. Measured on the 2026-10-27 deep dive: open-loop accuracy did not predict closed-loop driving.
- Closed-loop numbers come from a generative world model, not a car. In a separate test on the G55's windows, worldmodel-4B followed an added yaw command through 2 s but showed 0 to 20% of an added lateral offset 3 to 4 s later; past about 4 s the recorded anchor frames also pull every model back toward the recorded path. The front-page ranking uses 2 s.
- Intervals are 95% percentile bootstraps that resample whole drives (the drive a public segment was cut from, the recorded drive a truck window comes from), so segments of one drive count once. Below 10 drives there is no interval: the truck sections come from fewer drives and report means and win counts only.
- Paired deltas are model minus baseline on the units both have (for lateral gain, the change in its distance from 1); wins count units where the model is better in the metric's direction.
- The lead of the lateral request is the shift, searched from 0 to 1.5 s, at which the request best correlates with the car's later lateral motion; a larger value means the request comes earlier. A value at either edge is censored and left out. Plan speed error at 3 s reads the plan head, which does not separate models built on one frozen base: it is identical on every segment for the four f22e06e5 models, and for Cinque v2 and v3.
- Truck open loop pools few windows (camera mounts changed between drives); read it as a check that a model works on the truck's camera, not a ranking.
The model file
- Model id
870a4823_12864- SHA-256
c392ba1aa1b4b89e850366024fa6c1beac1f1fdc3a7182e3c3b0234c55e08eeb- File scored
- huggingface,
af05983f67bf9adb4df004dd8382d2aae5763ed4(link), 2026-10-08 - First public
- 2026-10-08
- Parameters
- 382.9M
- GMAC per step
- 50.1
- Family
- residual
- Scored
- 2026-10-27 21:27 UTC, suite
geval-1, code102710f59c - Notes
- Uploaded as "mountain dew model"; never shipped. Vision base f22e06e5 with an RL residual head.
Results file (results/870a4823_12864.json, schema 1).