Mountain Dew 8d736683
2026-10-10 Shipped in openpilot v0.11.2, on master from 2026-10-10
Lateral action rate 29% lower and lateral error 32% higher than lebowski (39 segments). Against 870a4823: higher lateral action rate, lateral lead and lateral error.
Key numbers
The tick is lebowski. Lateral action rate and lateral lead have no better direction; on the others lower is better. Compared with lebowski and with 870a4823, the model first public before it.
All scores
better, 95% interval excludes zero Better, 95% interval excludes zeroworse, 95% interval excludes zero Worse, interval excludes zerolower, 95% interval excludes zero; no better directionhigher, 95% interval excludes zero; no better direction Lower or higher, no better direction (action rate, lead)not separated, interval includes zero Not separatedno interval (too few drives) No interval: fewer than 10 drives
comma's public drives, open loop
39 segments from 23 drives. Per-segment CSV.
Measured. comma's public two-camera segments (openpilot CI data), native intrinsics and the device's own calibration, both cameras; every model on the same decoded frames. Truth is what the recording car did next: carState speed and acceleration, yaw rate from the device's cameraOdometry. Segments where the car never moves have no data. 39 of the 45 listed segments have a score, from 23 drives and 20 cars.
| Metric | Mean | vs lebowski | vs 870a4823 |
|---|---|---|---|
| Lateral errorm/s², lower is better | 0.143[0.113, 0.176]median 0.127 | worse, 95% interval excludes zero+0.035[+0.023, +0.047]better on 3/39 | worse, 95% interval excludes zero+0.010[+0.005, +0.015]better on 7/39 |
| Lateral error at the model's own best leadm/s², lower is better | 0.132[0.103, 0.164]median 0.114 | worse, 95% interval excludes zero+0.031[+0.018, +0.044]better on 7/39 | worse, 95% interval excludes zero+0.006[+0.001, +0.011]better on 13/39 |
| Lateral correlation with the car's later motionhigher is better | 0.821[0.726, 0.901]median 0.929 | worse, 95% interval excludes zero−0.037[−0.084, −0.003]better on 7/39 | worse, 95% interval excludes zero−0.020[−0.040, −0.004]better on 4/39 |
| Lateral gain (1 = same size as the car's motion)closer to 1 is better; the delta is the change in distance from 1 | 0.732[0.642, 0.826]median 0.739 | worse, 95% interval excludes zero+0.089[+0.026, +0.152]better on 11/39 | worse, 95% interval excludes zero+0.028[+0.008, +0.048]better on 13/39 |
| Lateral action rate, RMS (smoothness)m/s³, no better direction | 0.474[0.374, 0.595]median 0.422 | lower, 95% interval excludes zero; no better direction−0.190[−0.239, −0.134]lower on 35/39 | higher, 95% interval excludes zero; no better direction+0.036[+0.026, +0.053]lower on 2/39 |
| Lead of the lateral request over the car's motions, no better direction | 0.615[0.504, 0.736]median 0.500n = 36 | higher, 95% interval excludes zero; no better direction+0.226[+0.138, +0.328]lower on 1/35 | higher, 95% interval excludes zero; no better direction+0.101[+0.071, +0.146]lower on 0/36 |
| Longitudinal errorm/s², lower is better | 0.313[0.274, 0.358]median 0.294 | not separated, interval includes zero+0.005[−0.011, +0.023]better on 16/39 | not separated, interval includes zero+0.004[−0.001, +0.009]better on 14/39 |
| Longitudinal correlationhigher is better | 0.724[0.618, 0.817]median 0.849 | not separated, interval includes zero+0.006[−0.021, +0.031]better on 20/39 | not separated, interval includes zero−0.008[−0.020, +0.002]better on 14/39 |
| Longitudinal error while brakingm/s², lower is better | 0.427[0.367, 0.500]median 0.372n = 30 | not separated, interval includes zero−0.007[−0.030, +0.020]better on 13/30 | worse, 95% interval excludes zero+0.016[+0.007, +0.024]better on 10/30 |
| Longitudinal action rate, RMS (smoothness)m/s³, no better direction | 0.897[0.781, 1.038]median 0.768 | lower, 95% interval excludes zero; no better direction−0.436[−0.578, −0.312]lower on 36/39 | not separated, interval includes zero+0.003[−0.014, +0.020]lower on 20/39 |
| Plan speed error at 3 sm/s, lower is better | 0.760[0.657, 0.888]median 0.712 | worse, 95% interval excludes zero+0.044[+0.009, +0.077]better on 12/39 | not separated, interval includes zeroidenticalon all 39 |
comma's public drives, closed loop
28 segments from 19 drives. Per-segment CSV.
Measured. comma's worldmodel-4B generates each next frame at 5 Hz from 9 history frames, 5 recorded anchor frames and the car's pose; the model drives comma's ideal car (the requested curvature and acceleration happen at once) for 7.2 s from the recorded start. Distance is to the recording car's own path. One window per segment, nearest the middle, where the car holds at least 5 m/s for the window and the 4.8 s before it; segments with no such stretch are left out. Seeds 0 and 1, averaged per segment. 28 of the 45 listed segments have a score, from 19 drives and 17 cars.
| Metric | Mean | vs lebowski |
|---|---|---|
| Distance from the recorded path at 2 sm, lower is better | 0.096[0.048, 0.182]median 0.056 | better, 95% interval excludes zero−0.051[−0.102, −0.014]better on 19/28 |
| Distance from the recorded path at 4 sm, lower is better | 0.221[0.103, 0.407]median 0.109 | better, 95% interval excludes zero−0.057[−0.126, −0.001]better on 11/28 |
| Distance from the recorded path at 7.2 sm, lower is better | 1.02[0.34, 1.90]median 0.31 | not separated, interval includes zero−0.02[−0.59, +0.53]better on 15/28 |
| Share of episodes ever more than 1 m off the pathlower is better | 0.179[0.038, 0.333]median 0.000 | not separated, interval includes zero−0.054[−0.152, 0.000]better on 2/28 |
| Requested lateral jerk, RMSm/s³, lower is better | 0.202[0.136, 0.272]median 0.119 | better, 95% interval excludes zero−0.123[−0.179, −0.069]better on 24/28 |
| Requested longitudinal jerk, RMSm/s³, lower is better | 0.341[0.262, 0.433]median 0.289 | not separated, interval includes zero−0.005[−0.045, +0.042]better on 17/28 |
| Speed error against the recorded speed, RMSm/s, lower is better | 1.24[0.87, 1.64]median 0.86 | not separated, interval includes zero+0.14[−0.09, +0.36]better on 8/28 |
G55, open loop
4 windows from 3 drives.
Measured. The G55's recorded drives replayed as the truck runs the model: narrow camera only, road-plane warp to 1.45 m (plane:1.45). Truth is the truck's CAN speed and yaw rate. Window = unit. 4 windows: 3 on an earlier camera mount, 1 on the current mount; 2 at night.
| Metric | Mean | vs lebowski | vs 870a4823 |
|---|---|---|---|
| Lateral errorm/s², lower is better | 0.150no intervalmedian 0.164 | no interval (too few drives)+0.023no intervalbetter on 0/40/3 drives | no interval (too few drives)+0.009no intervalbetter on 0/40/3 drives |
| Lateral error at the model's own best leadm/s², lower is better | 0.139no intervalmedian 0.146 | no interval (too few drives)+0.012no intervalbetter on 0/40/3 drives | no interval (too few drives)+0.005no intervalbetter on 0/40/3 drives |
| Lateral correlation with the car's later motionhigher is better | 0.937no intervalmedian 0.943 | no interval (too few drives)−0.023no intervalbetter on 0/40/3 drives | no interval (too few drives)−0.006no intervalbetter on 0/40/3 drives |
| Lateral gain (1 = same size as the car's motion)closer to 1 is better; the delta is the change in distance from 1 | 0.809no intervalmedian 0.804 | no interval (too few drives)+0.133no intervalbetter on 0/40/3 drives | no interval (too few drives)+0.032no intervalbetter on 0/40/3 drives |
| Lateral action rate, RMS (smoothness)m/s³, no better direction | 0.547no intervalmedian 0.568 | no interval (too few drives)−0.410no intervallower on 4/43/3 drives | no interval (too few drives)+0.020no intervallower on 0/40/3 drives |
| Lead of the lateral request over the car's motions, no better direction | 0.525no intervalmedian 0.575 | no interval (too few drives)+0.188no intervallower on 0/40/3 drives | no interval (too few drives)+0.062no intervallower on 0/40/3 drives |
| Longitudinal errorm/s², lower is better | 0.323no intervalmedian 0.327 | no interval (too few drives)+0.008no intervalbetter on 1/40/3 drives | no interval (too few drives)−0.004no intervalbetter on 3/42/3 drives |
| Longitudinal correlationhigher is better | 0.533no intervalmedian 0.575 | no interval (too few drives)−0.049no intervalbetter on 0/40/3 drives | no interval (too few drives)−0.010no intervalbetter on 3/42/3 drives |
| Longitudinal error while brakingm/s², lower is better | 0.560no intervalmedian 0.547 | no interval (too few drives)+0.045no intervalbetter on 0/40/3 drives | no interval (too few drives)+0.024no intervalbetter on 1/41/3 drives |
| Longitudinal action rate, RMS (smoothness)m/s³, no better direction | 1.73no intervalmedian 1.29 | no interval (too few drives)−0.02no intervallower on 3/42/3 drives | no interval (too few drives)+0.05no intervallower on 2/41/3 drives |
| Plan speed error at 3 sm/s, lower is better | 1.30no intervalmedian 1.03 | no interval (too few drives)+0.28no intervalbetter on 1/41/3 drives | not separated, interval includes zeroidenticalon all 4 |
G55, closed loop
18 windows from 4 drives.
Measured. worldmodel-4B as in the public closed loop, but the model drives a simulation of the G55's measured steering and powertrain response instead of the ideal car. Lane-following windows from the G55's drives, day and night; distance is to the driver's recorded path; seeds 0 and 1, averaged per window. 18 windows (9 day, 9 night).
| Metric | Mean | vs lebowski |
|---|---|---|
| Distance from the recorded path at 2 sm, lower is better | 0.229no intervalmedian 0.159 | no interval (too few drives)−0.039no intervalbetter on 13/184/4 drives |
| Distance from the recorded path at 4 sm, lower is better | 0.526no intervalmedian 0.441 | no interval (too few drives)−0.036no intervalbetter on 7/183/4 drives |
| Distance from the recorded path at 7.2 sm, lower is better | 1.51no intervalmedian 0.85 | no interval (too few drives)−0.10no intervalbetter on 7/181/4 drives |
| Share of episodes ever more than 1 m off the pathlower is better | 0.472no intervalmedian 0.500 | no interval (too few drives)+0.111no intervalbetter on 2/181/4 drives |
| Requested lateral jerk, RMSm/s³, lower is better | 0.411no intervalmedian 0.205 | no interval (too few drives)−0.134no intervalbetter on 16/184/4 drives |
| Requested longitudinal jerk, RMSm/s³, lower is better | 0.386no intervalmedian 0.346 | no interval (too few drives)+0.025no intervalbetter on 8/180/4 drives |
| Speed error against the recorded speed, RMSm/s, lower is better | 1.03no intervalmedian 0.93 | no interval (too few drives)−0.02no intervalbetter on 8/183/4 drives |
G55, recovery tests
18 windows from 4 drives.
Measured. The truck closed loop with a disturbance while the model drives; only the tests listed ran. 2° off heading: an extra 2°/s yaw for the first second. 18 windows (9 day, 9 night). 0.5 m off centre: an extra yaw-rate pulse, out and back over the first 2 s, that would move an uncorrected car 0.5 m sideways with no net heading change. 18 windows (9 day, 9 night).
| Metric | Mean | vs lebowski |
|---|---|---|
| 2° off heading: distance from the path at 2 sm, lower is better | 0.411no intervalmedian 0.341 | no interval (too few drives)−0.016no intervalbetter on 10/182/4 drives |
| 2° off heading: distance from the path at 4 sm, lower is better | 0.573no intervalmedian 0.490 | no interval (too few drives)+0.029no intervalbetter on 7/182/4 drives |
| 2° off heading: requested lateral jerk, RMSm/s³, lower is better | 0.448no intervalmedian 0.286 | no interval (too few drives)−0.379no intervalbetter on 15/184/4 drives |
| 0.5 m off centre: distance from the path at 2 sm, lower is better | 0.338no intervalmedian 0.285 | no interval (too few drives)−0.011no intervalbetter on 10/184/4 drives |
| 0.5 m off centre: distance from the path at 4 sm, lower is better | 0.530no intervalmedian 0.488 | no interval (too few drives)−0.129no intervalbetter on 12/183/4 drives |
| 0.5 m off centre: requested lateral jerk, RMSm/s³, lower is better | 0.595no intervalmedian 0.455 | no interval (too few drives)−0.578no intervalbetter on 16/184/4 drives |
Limits
- Open-loop scores compare a model's action with what the recording driver or car did; a model that drives differently but well is scored as wrong. comma's RL residual makes the lateral action smoother, and earlier than its base's by 0.10 to 0.26 s (measured on the G55 and on 16 of comma's cars); both score as error here. Measured on the 2026-10-27 deep dive: open-loop accuracy did not predict closed-loop driving.
- Closed-loop numbers come from a generative world model, not a car. In a separate test on the G55's windows, worldmodel-4B followed an added yaw command through 2 s but showed 0 to 20% of an added lateral offset 3 to 4 s later; past about 4 s the recorded anchor frames also pull every model back toward the recorded path. The front-page ranking uses 2 s.
- Intervals are 95% percentile bootstraps that resample whole drives (the drive a public segment was cut from, the recorded drive a truck window comes from), so segments of one drive count once. Below 10 drives there is no interval: the truck sections come from fewer drives and report means and win counts only.
- Paired deltas are model minus baseline on the units both have (for lateral gain, the change in its distance from 1); wins count units where the model is better in the metric's direction.
- The lead of the lateral request is the shift, searched from 0 to 1.5 s, at which the request best correlates with the car's later lateral motion; a larger value means the request comes earlier. A value at either edge is censored and left out. Plan speed error at 3 s reads the plan head, which does not separate models built on one frozen base: it is identical on every segment for the four f22e06e5 models, and for Cinque v2 and v3.
- Truck open loop pools few windows (camera mounts changed between drives); read it as a check that a model works on the truck's camera, not a ranking.
- Truck closed loop: the steering and powertrain simulation is fitted to the G55, not validated on it in closed loop; the reference is the driver's own line, not the lane centre.
- Recovery tests are disturbances rendered by the world model (yaw it renders; a one-frame jump it does not, so none is used). The braking-lead test is not built yet: the drives the truck closed loop can use hold no window where a car ahead brakes.
The model file
- Model id
8d736683_12864- SHA-256
87fe86ca0813c099efddf9173ba503be2e40ea4c5362c36812a44141f4cb07d4- File scored
- release,
v0.11.2(link), 2026-10-23 - First public
- 2026-10-10
- Parameters
- 382.9M
- GMAC per step
- 50.1
- Family
- residual
- Scored
- 2026-10-27 21:27 UTC, suite
geval-1, code102710f59c - Notes
- Shipped in openpilot 0.11.2 (master 2026-10-10, PR #39047). Scored from the Hugging Face upload of 2026-10-14, whose tensors match the release's compiled model byte for byte. Vision base f22e06e5 with an RL residual head.
Results file (results/8d736683_12864.json, schema 1).