MD v3 / Flip Flop 1365126a
2026-10-14 Merged to openpilot master 2026-10-14, again 2026-10-24
Lateral action rate 32% lower and lateral error 18% higher than lebowski (39 segments). Against Mountain Dew: lower lateral action rate, lateral lead, lateral error and longitudinal error.
Key numbers
The tick is lebowski. Lateral action rate and lateral lead have no better direction; on the others lower is better. Compared with lebowski and with Mountain Dew 8d736683, the model first public before it.
All scores
better, 95% interval excludes zero Better, 95% interval excludes zeroworse, 95% interval excludes zero Worse, interval excludes zerolower, 95% interval excludes zero; no better directionhigher, 95% interval excludes zero; no better direction Lower or higher, no better direction (action rate, lead)not separated, interval includes zero Not separatedno interval (too few drives) No interval: fewer than 10 drives
comma's public drives, open loop
39 segments from 23 drives. Per-segment CSV.
Measured. comma's public two-camera segments (openpilot CI data), native intrinsics and the device's own calibration, both cameras; every model on the same decoded frames. Truth is what the recording car did next: carState speed and acceleration, yaw rate from the device's cameraOdometry. Segments where the car never moves have no data. 39 of the 45 listed segments have a score, from 23 drives and 20 cars.
| Metric | Mean | vs lebowski | vs Mountain Dew |
|---|---|---|---|
| Lateral errorm/s², lower is better | 0.129[0.101, 0.159]median 0.110 | worse, 95% interval excludes zero+0.020[+0.010, +0.032]better on 10/39 | better, 95% interval excludes zero−0.015[−0.020, −0.009]better on 36/39 |
| Lateral error at the model's own best leadm/s², lower is better | 0.119[0.093, 0.148]median 0.098 | worse, 95% interval excludes zero+0.018[+0.007, +0.029]better on 10/39 | better, 95% interval excludes zero−0.013[−0.018, −0.007]better on 36/39 |
| Lateral correlation with the car's later motionhigher is better | 0.847[0.759, 0.921]median 0.939 | not separated, interval includes zero−0.011[−0.053, +0.022]better on 12/39 | better, 95% interval excludes zero+0.026[+0.013, +0.042]better on 35/39 |
| Lateral gain (1 = same size as the car's motion)closer to 1 is better; the delta is the change in distance from 1 | 0.756[0.666, 0.851]median 0.795 | worse, 95% interval excludes zero+0.064[+0.012, +0.117]better on 13/39 | better, 95% interval excludes zero−0.024[−0.046, −0.002]better on 32/39 |
| Lateral action rate, RMS (smoothness)m/s³, no better direction | 0.451[0.353, 0.566]median 0.397 | lower, 95% interval excludes zero; no better direction−0.212[−0.256, −0.166]lower on 38/39 | lower, 95% interval excludes zero; no better direction−0.023[−0.040, −0.011]lower on 32/39 |
| Lead of the lateral request over the car's motions, no better direction | 0.470[0.377, 0.574]median 0.450n = 35 | higher, 95% interval excludes zero; no better direction+0.088[+0.035, +0.152]lower on 7/34 | lower, 95% interval excludes zero; no better direction−0.127[−0.191, −0.077]lower on 30/35 |
| Longitudinal errorm/s², lower is better | 0.301[0.261, 0.347]median 0.268 | not separated, interval includes zero−0.007[−0.023, +0.010]better on 19/39 | better, 95% interval excludes zero−0.012[−0.019, −0.005]better on 30/39 |
| Longitudinal correlationhigher is better | 0.739[0.640, 0.824]median 0.850 | not separated, interval includes zero+0.021[−0.004, +0.045]better on 24/39 | better, 95% interval excludes zero+0.015[0.000, +0.027]better on 31/39 |
| Longitudinal error while brakingm/s², lower is better | 0.401[0.340, 0.471]median 0.366n = 30 | better, 95% interval excludes zero−0.034[−0.056, −0.011]better on 20/30 | better, 95% interval excludes zero−0.027[−0.040, −0.015]better on 24/30 |
| Longitudinal action rate, RMS (smoothness)m/s³, no better direction | 0.890[0.780, 1.020]median 0.773 | lower, 95% interval excludes zero; no better direction−0.443[−0.574, −0.329]lower on 37/39 | not separated, interval includes zero−0.007[−0.032, +0.015]lower on 21/39 |
| Plan speed error at 3 sm/s, lower is better | 0.760[0.657, 0.888]median 0.712 | worse, 95% interval excludes zero+0.044[+0.009, +0.077]better on 12/39 | not separated, interval includes zeroidenticalon all 39 |
comma's public drives, closed loop
28 segments from 19 drives. Per-segment CSV.
Measured. comma's worldmodel-4B generates each next frame at 5 Hz from 9 history frames, 5 recorded anchor frames and the car's pose; the model drives comma's ideal car (the requested curvature and acceleration happen at once) for 7.2 s from the recorded start. Distance is to the recording car's own path. One window per segment, nearest the middle, where the car holds at least 5 m/s for the window and the 4.8 s before it; segments with no such stretch are left out. Seeds 0 and 1, averaged per segment. 28 of the 45 listed segments have a score, from 19 drives and 17 cars.
| Metric | Mean | vs lebowski | vs Mountain Dew |
|---|---|---|---|
| Distance from the recorded path at 2 sm, lower is better | 0.087[0.056, 0.135]median 0.074 | better, 95% interval excludes zero−0.060[−0.147, −0.006]better on 17/28 | not separated, interval includes zero−0.009[−0.048, +0.014]better on 11/28 |
| Distance from the recorded path at 4 sm, lower is better | 0.170[0.114, 0.233]median 0.127 | not separated, interval includes zero−0.108[−0.311, +0.015]better on 12/28 | not separated, interval includes zero−0.051[−0.204, +0.039]better on 12/28 |
| Distance from the recorded path at 7.2 sm, lower is better | 0.999[0.333, 1.900]median 0.303 | not separated, interval includes zero−0.045[−0.621, +0.527]better on 10/28 | not separated, interval includes zero−0.026[−0.174, +0.087]better on 11/28 |
| Share of episodes ever more than 1 m off the pathlower is better | 0.143[0.000, 0.300]median 0.000 | not separated, interval includes zero−0.089[−0.205, 0.000]better on 3/28 | not separated, interval includes zero−0.036[−0.111, 0.000]better on 1/28 |
| Requested lateral jerk, RMSm/s³, lower is better | 0.201[0.126, 0.285]median 0.096 | better, 95% interval excludes zero−0.125[−0.183, −0.064]better on 25/28 | not separated, interval includes zero−0.001[−0.025, +0.029]better on 19/28 |
| Requested longitudinal jerk, RMSm/s³, lower is better | 0.356[0.275, 0.455]median 0.301 | not separated, interval includes zero+0.010[−0.030, +0.060]better on 15/28 | not separated, interval includes zero+0.015[−0.002, +0.032]better on 9/28 |
| Speed error against the recorded speed, RMSm/s, lower is better | 1.18[0.84, 1.56]median 0.70 | not separated, interval includes zero+0.07[−0.13, +0.27]better on 8/28 | not separated, interval includes zero−0.07[−0.13, 0.00]better on 17/28 |
G55, open loop
4 windows from 3 drives.
Measured. The G55's recorded drives replayed as the truck runs the model: narrow camera only, road-plane warp to 1.45 m (plane:1.45). Truth is the truck's CAN speed and yaw rate. Window = unit. 4 windows: 3 on an earlier camera mount, 1 on the current mount; 2 at night.
| Metric | Mean | vs lebowski | vs Mountain Dew |
|---|---|---|---|
| Lateral errorm/s², lower is better | 0.142no intervalmedian 0.157 | no interval (too few drives)+0.014no intervalbetter on 1/41/3 drives | no interval (too few drives)−0.009no intervalbetter on 4/43/3 drives |
| Lateral error at the model's own best leadm/s², lower is better | 0.133no intervalmedian 0.144 | no interval (too few drives)+0.006no intervalbetter on 2/41/3 drives | no interval (too few drives)−0.006no intervalbetter on 4/43/3 drives |
| Lateral correlation with the car's later motionhigher is better | 0.941no intervalmedian 0.948 | no interval (too few drives)−0.019no intervalbetter on 1/41/3 drives | no interval (too few drives)+0.004no intervalbetter on 4/43/3 drives |
| Lateral gain (1 = same size as the car's motion)closer to 1 is better; the delta is the change in distance from 1 | 0.829no intervalmedian 0.842 | no interval (too few drives)+0.113no intervalbetter on 0/40/3 drives | no interval (too few drives)−0.019no intervalbetter on 3/43/3 drives |
| Lateral action rate, RMS (smoothness)m/s³, no better direction | 0.544no intervalmedian 0.574 | no interval (too few drives)−0.412no intervallower on 4/43/3 drives | no interval (too few drives)−0.002no intervallower on 2/41/3 drives |
| Lead of the lateral request over the car's motions, no better direction | 0.475no intervalmedian 0.550 | no interval (too few drives)+0.138no intervallower on 1/41/3 drives | no interval (too few drives)−0.050no intervallower on 3/43/3 drives |
| Longitudinal errorm/s², lower is better | 0.332no intervalmedian 0.334 | no interval (too few drives)+0.017no intervalbetter on 0/40/3 drives | no interval (too few drives)+0.009no intervalbetter on 2/41/3 drives |
| Longitudinal correlationhigher is better | 0.545no intervalmedian 0.583 | no interval (too few drives)−0.037no intervalbetter on 1/41/3 drives | no interval (too few drives)+0.012no intervalbetter on 2/42/3 drives |
| Longitudinal error while brakingm/s², lower is better | 0.540no intervalmedian 0.549 | no interval (too few drives)+0.024no intervalbetter on 1/41/3 drives | no interval (too few drives)−0.021no intervalbetter on 3/42/3 drives |
| Longitudinal action rate, RMS (smoothness)m/s³, no better direction | 1.74no intervalmedian 1.31 | no interval (too few drives)−0.01no intervallower on 3/42/3 drives | no interval (too few drives)0.00no intervallower on 2/41/3 drives |
| Plan speed error at 3 sm/s, lower is better | 1.30no intervalmedian 1.03 | no interval (too few drives)+0.28no intervalbetter on 1/41/3 drives | not separated, interval includes zeroidenticalon all 4 |
G55, closed loop
18 windows from 4 drives.
Measured. worldmodel-4B as in the public closed loop, but the model drives a simulation of the G55's measured steering and powertrain response instead of the ideal car. Lane-following windows from the G55's drives, day and night; distance is to the driver's recorded path; seeds 0 and 1, averaged per window. 18 windows (9 day, 9 night).
| Metric | Mean | vs lebowski | vs Mountain Dew |
|---|---|---|---|
| Distance from the recorded path at 2 sm, lower is better | 0.211no intervalmedian 0.162 | no interval (too few drives)−0.057no intervalbetter on 12/184/4 drives | no interval (too few drives)−0.018no intervalbetter on 9/183/4 drives |
| Distance from the recorded path at 4 sm, lower is better | 0.512no intervalmedian 0.463 | no interval (too few drives)−0.050no intervalbetter on 8/182/4 drives | no interval (too few drives)−0.013no intervalbetter on 8/182/4 drives |
| Distance from the recorded path at 7.2 sm, lower is better | 1.76no intervalmedian 1.01 | no interval (too few drives)+0.15no intervalbetter on 5/181/4 drives | no interval (too few drives)+0.25no intervalbetter on 6/180/4 drives |
| Share of episodes ever more than 1 m off the pathlower is better | 0.500no intervalmedian 0.500 | no interval (too few drives)+0.139no intervalbetter on 2/182/4 drives | no interval (too few drives)+0.028no intervalbetter on 1/181/4 drives |
| Requested lateral jerk, RMSm/s³, lower is better | 0.382no intervalmedian 0.198 | no interval (too few drives)−0.162no intervalbetter on 16/184/4 drives | no interval (too few drives)−0.028no intervalbetter on 12/182/4 drives |
| Requested longitudinal jerk, RMSm/s³, lower is better | 0.384no intervalmedian 0.357 | no interval (too few drives)+0.023no intervalbetter on 10/182/4 drives | no interval (too few drives)−0.002no intervalbetter on 9/182/4 drives |
| Speed error against the recorded speed, RMSm/s, lower is better | 1.05no intervalmedian 1.00 | no interval (too few drives)−0.01no intervalbetter on 11/182/4 drives | no interval (too few drives)+0.01no intervalbetter on 8/181/4 drives |
G55, recovery tests
18 windows from 4 drives.
Measured. The truck closed loop with a disturbance while the model drives; only the tests listed ran. 2° off heading: an extra 2°/s yaw for the first second. 18 windows (9 day, 9 night). 0.5 m off centre: an extra yaw-rate pulse, out and back over the first 2 s, that would move an uncorrected car 0.5 m sideways with no net heading change. 18 windows (9 day, 9 night).
| Metric | Mean | vs lebowski | vs Mountain Dew |
|---|---|---|---|
| 2° off heading: distance from the path at 2 sm, lower is better | 0.421no intervalmedian 0.380 | no interval (too few drives)−0.006no intervalbetter on 8/183/4 drives | no interval (too few drives)+0.010no intervalbetter on 2/181/4 drives |
| 2° off heading: distance from the path at 4 sm, lower is better | 0.559no intervalmedian 0.526 | no interval (too few drives)+0.015no intervalbetter on 5/182/4 drives | no interval (too few drives)−0.014no intervalbetter on 9/183/4 drives |
| 2° off heading: requested lateral jerk, RMSm/s³, lower is better | 0.408no intervalmedian 0.299 | no interval (too few drives)−0.419no intervalbetter on 15/184/4 drives | no interval (too few drives)−0.040no intervalbetter on 10/183/4 drives |
| 0.5 m off centre: distance from the path at 2 sm, lower is better | 0.341no intervalmedian 0.295 | no interval (too few drives)−0.008no intervalbetter on 7/182/4 drives | no interval (too few drives)+0.003no intervalbetter on 6/181/4 drives |
| 0.5 m off centre: distance from the path at 4 sm, lower is better | 0.498no intervalmedian 0.390 | no interval (too few drives)−0.161no intervalbetter on 13/183/4 drives | no interval (too few drives)−0.032no intervalbetter on 10/183/4 drives |
| 0.5 m off centre: requested lateral jerk, RMSm/s³, lower is better | 0.685no intervalmedian 0.448 | no interval (too few drives)−0.488no intervalbetter on 15/184/4 drives | no interval (too few drives)+0.090no intervalbetter on 7/182/4 drives |
Limits
- Open-loop scores compare a model's action with what the recording driver or car did; a model that drives differently but well is scored as wrong. comma's RL residual makes the lateral action smoother, and earlier than its base's by 0.10 to 0.26 s (measured on the G55 and on 16 of comma's cars); both score as error here. Measured on the 2026-10-27 deep dive: open-loop accuracy did not predict closed-loop driving.
- Closed-loop numbers come from a generative world model, not a car. In a separate test on the G55's windows, worldmodel-4B followed an added yaw command through 2 s but showed 0 to 20% of an added lateral offset 3 to 4 s later; past about 4 s the recorded anchor frames also pull every model back toward the recorded path. The front-page ranking uses 2 s.
- Intervals are 95% percentile bootstraps that resample whole drives (the drive a public segment was cut from, the recorded drive a truck window comes from), so segments of one drive count once. Below 10 drives there is no interval: the truck sections come from fewer drives and report means and win counts only.
- Paired deltas are model minus baseline on the units both have (for lateral gain, the change in its distance from 1); wins count units where the model is better in the metric's direction.
- The lead of the lateral request is the shift, searched from 0 to 1.5 s, at which the request best correlates with the car's later lateral motion; a larger value means the request comes earlier. A value at either edge is censored and left out. Plan speed error at 3 s reads the plan head, which does not separate models built on one frozen base: it is identical on every segment for the four f22e06e5 models, and for Cinque v2 and v3.
- Truck open loop pools few windows (camera mounts changed between drives); read it as a check that a model works on the truck's camera, not a ranking.
- Truck closed loop: the steering and powertrain simulation is fitted to the G55, not validated on it in closed loop; the reference is the driver's own line, not the lane centre.
- Recovery tests are disturbances rendered by the world model (yaw it renders; a one-frame jump it does not, so none is used). The braking-lead test is not built yet: the drives the truck closed loop can use hold no window where a car ahead brakes.
The model file
- Model id
1365126a_12864- SHA-256
35976b11be3904eb78e32f7dc55a6abf8e9433ac5aa391a1998f1e9ce001b199- File scored
- openpilot-master,
9fba666fd4(link), 2026-10-24 - First public
- 2026-10-14
- Parameters
- 382.9M
- GMAC per step
- 46.7
- Family
- residual
- Scored
- 2026-10-27 21:27 UTC, suite
geval-1, code102710f59c - Notes
- On openpilot master as MD v3 (2026-10-14, PR #39087) and again as Flip Flop (2026-10-24, PR #39156), same weights. Scored from the 2026-10-24 Hugging Face re-upload, weights identical to the 2026-10-16 upload. Vision base f22e06e5 with an RL residual head.
Results file (results/1365126a_12864.json, schema 1).