Cinque v2 1a421175
2026-09-11 Merged to openpilot master 2026-09-11
Lateral action rate 38% lower and lateral error 36% higher than lebowski (39 segments).
Key numbers
The tick is lebowski. Lateral action rate and lateral lead have no better direction; on the others lower is better.
All scores
better, 95% interval excludes zero Better, 95% interval excludes zeroworse, 95% interval excludes zero Worse, interval excludes zerolower, 95% interval excludes zero; no better directionhigher, 95% interval excludes zero; no better direction Lower or higher, no better direction (action rate, lead)not separated, interval includes zero Not separatedno interval (too few drives) No interval: fewer than 10 drives
comma's public drives, open loop
39 segments from 23 drives. Per-segment CSV.
Measured. comma's public two-camera segments (openpilot CI data), native intrinsics and the device's own calibration, both cameras; every model on the same decoded frames. Truth is what the recording car did next: carState speed and acceleration, yaw rate from the device's cameraOdometry. Segments where the car never moves have no data. 39 of the 45 listed segments have a score, from 23 drives and 20 cars.
| Metric | Mean | vs lebowski |
|---|---|---|
| Lateral errorm/s², lower is better | 0.147[0.113, 0.183]median 0.120 | worse, 95% interval excludes zero+0.039[+0.020, +0.058]better on 7/39 |
| Lateral error at the model's own best leadm/s², lower is better | 0.138[0.104, 0.173]median 0.118 | worse, 95% interval excludes zero+0.036[+0.017, +0.057]better on 10/39 |
| Lateral correlation with the car's later motionhigher is better | 0.795[0.698, 0.883]median 0.928 | worse, 95% interval excludes zero−0.062[−0.135, −0.003]better on 10/39 |
| Lateral gain (1 = same size as the car's motion)closer to 1 is better; the delta is the change in distance from 1 | 0.702[0.600, 0.806]median 0.758 | worse, 95% interval excludes zero+0.118[+0.025, +0.214]better on 11/39 |
| Lateral action rate, RMS (smoothness)m/s³, no better direction | 0.412[0.336, 0.498]median 0.351 | lower, 95% interval excludes zero; no better direction−0.252[−0.304, −0.198]lower on 38/39 |
| Lead of the lateral request over the car's motions, no better direction | 0.492[0.383, 0.603]median 0.450n = 32 | higher, 95% interval excludes zero; no better direction+0.092[+0.031, +0.158]lower on 6/31 |
| Longitudinal errorm/s², lower is better | 0.317[0.272, 0.367]median 0.284 | not separated, interval includes zero+0.009[−0.006, +0.025]better on 16/39 |
| Longitudinal correlationhigher is better | 0.717[0.608, 0.811]median 0.835 | not separated, interval includes zero0.000[−0.026, +0.024]better on 21/39 |
| Longitudinal error while brakingm/s², lower is better | 0.441[0.373, 0.522]median 0.421n = 30 | not separated, interval includes zero+0.006[−0.023, +0.043]better on 11/30 |
| Longitudinal action rate, RMS (smoothness)m/s³, no better direction | 0.768[0.694, 0.852]median 0.660 | lower, 95% interval excludes zero; no better direction−0.565[−0.717, −0.424]lower on 37/39 |
| Plan speed error at 3 sm/s, lower is better | 0.758[0.658, 0.888]median 0.744 | not separated, interval includes zero+0.043[−0.010, +0.094]better on 14/39 |
comma's public drives, closed loop
Not run for this model yet.
G55, open loop
4 windows from 3 drives.
Measured. The G55's recorded drives replayed as the truck runs the model: narrow camera only, road-plane warp to 1.45 m (plane:1.45). Truth is the truck's CAN speed and yaw rate. Window = unit. 4 windows: 3 on an earlier camera mount, 1 on the current mount; 2 at night.
| Metric | Mean | vs lebowski |
|---|---|---|
| Lateral errorm/s², lower is better | 0.151no intervalmedian 0.163 | no interval (too few drives)+0.024no intervalbetter on 0/40/3 drives |
| Lateral error at the model's own best leadm/s², lower is better | 0.142no intervalmedian 0.150 | no interval (too few drives)+0.016no intervalbetter on 1/41/3 drives |
| Lateral correlation with the car's later motionhigher is better | 0.930no intervalmedian 0.937 | no interval (too few drives)−0.030no intervalbetter on 0/40/3 drives |
| Lateral gain (1 = same size as the car's motion)closer to 1 is better; the delta is the change in distance from 1 | 0.824no intervalmedian 0.844 | no interval (too few drives)+0.118no intervalbetter on 0/40/3 drives |
| Lateral action rate, RMS (smoothness)m/s³, no better direction | 0.518no intervalmedian 0.524 | no interval (too few drives)−0.438no intervallower on 4/43/3 drives |
| Lead of the lateral request over the car's motions, no better direction | 0.463no intervalmedian 0.525 | no interval (too few drives)+0.125no intervallower on 1/41/3 drives |
| Longitudinal errorm/s², lower is better | 0.316no intervalmedian 0.313 | no interval (too few drives)+0.001no intervalbetter on 3/42/3 drives |
| Longitudinal correlationhigher is better | 0.562no intervalmedian 0.622 | no interval (too few drives)−0.020no intervalbetter on 0/40/3 drives |
| Longitudinal error while brakingm/s², lower is better | 0.611no intervalmedian 0.618 | no interval (too few drives)+0.096no intervalbetter on 0/40/3 drives |
| Longitudinal action rate, RMS (smoothness)m/s³, no better direction | 1.08no intervalmedian 1.03 | no interval (too few drives)−0.67no intervallower on 4/43/3 drives |
| Plan speed error at 3 sm/s, lower is better | 1.05no intervalmedian 1.05 | no interval (too few drives)+0.02no intervalbetter on 2/42/3 drives |
G55, closed loop
Not run for this model yet.
G55, recovery tests
Not run for this model yet.
Limits
- Open-loop scores compare a model's action with what the recording driver or car did; a model that drives differently but well is scored as wrong. comma's RL residual makes the lateral action smoother, and earlier than its base's by 0.10 to 0.26 s (measured on the G55 and on 16 of comma's cars); both score as error here. Measured on the 2026-10-27 deep dive: open-loop accuracy did not predict closed-loop driving.
- Closed-loop numbers come from a generative world model, not a car. In a separate test on the G55's windows, worldmodel-4B followed an added yaw command through 2 s but showed 0 to 20% of an added lateral offset 3 to 4 s later; past about 4 s the recorded anchor frames also pull every model back toward the recorded path. The front-page ranking uses 2 s.
- Intervals are 95% percentile bootstraps that resample whole drives (the drive a public segment was cut from, the recorded drive a truck window comes from), so segments of one drive count once. Below 10 drives there is no interval: the truck sections come from fewer drives and report means and win counts only.
- Paired deltas are model minus baseline on the units both have (for lateral gain, the change in its distance from 1); wins count units where the model is better in the metric's direction.
- The lead of the lateral request is the shift, searched from 0 to 1.5 s, at which the request best correlates with the car's later lateral motion; a larger value means the request comes earlier. A value at either edge is censored and left out. Plan speed error at 3 s reads the plan head, which does not separate models built on one frozen base: it is identical on every segment for the four f22e06e5 models, and for Cinque v2 and v3.
- Truck open loop pools few windows (camera mounts changed between drives); read it as a check that a model works on the truck's camera, not a ranking.
The model file
- Model id
1a421175_12864- SHA-256
c7b300083b8dd6ce17545ef52edb6b577e3a805e6372f9f9343090604690242f- File scored
- huggingface,
c83630ca0ea166f74c89a67bbdf2a3a820126de6(link), 2026-09-18 - First public
- 2026-09-11
- Parameters
- 381.6M
- GMAC per step
- 46.7
- Family
- big-rl
- Scored
- 2026-10-27 21:27 UTC, suite
geval-1, code102710f59c - Notes
- On openpilot master 2026-09-11 (PR #38850). Vision base b9facbcc, history in the graph, no residual head.
Results file (results/1a421175_12864.json, schema 1).