Cinque v3 f78ed37d
2026-09-18 Merged to openpilot master 2026-09-18
Lateral action rate 36% lower and lateral error 26% higher than lebowski (39 segments). Against Cinque v2: lower lateral error and longitudinal error; higher lateral action rate.
Key numbers
The tick is lebowski. Lateral action rate and lateral lead have no better direction; on the others lower is better. Compared with lebowski and with Cinque v2 1a421175, the model first public before it.
All scores
better, 95% interval excludes zero Better, 95% interval excludes zeroworse, 95% interval excludes zero Worse, interval excludes zerolower, 95% interval excludes zero; no better directionhigher, 95% interval excludes zero; no better direction Lower or higher, no better direction (action rate, lead)not separated, interval includes zero Not separatedno interval (too few drives) No interval: fewer than 10 drives
comma's public drives, open loop
39 segments from 23 drives. Per-segment CSV.
Measured. comma's public two-camera segments (openpilot CI data), native intrinsics and the device's own calibration, both cameras; every model on the same decoded frames. Truth is what the recording car did next: carState speed and acceleration, yaw rate from the device's cameraOdometry. Segments where the car never moves have no data. 39 of the 45 listed segments have a score, from 23 drives and 20 cars.
| Metric | Mean | vs lebowski | vs Cinque v2 |
|---|---|---|---|
| Lateral errorm/s², lower is better | 0.137[0.108, 0.166]median 0.124 | worse, 95% interval excludes zero+0.028[+0.017, +0.039]better on 7/39 | better, 95% interval excludes zero−0.011[−0.020, −0.003]better on 24/39 |
| Lateral error at the model's own best leadm/s², lower is better | 0.128[0.101, 0.157]median 0.115 | worse, 95% interval excludes zero+0.027[+0.015, +0.038]better on 8/39 | better, 95% interval excludes zero−0.009[−0.019, −0.001]better on 21/39 |
| Lateral correlation with the car's later motionhigher is better | 0.827[0.739, 0.905]median 0.940 | not separated, interval includes zero−0.031[−0.081, +0.011]better on 8/39 | better, 95% interval excludes zero+0.032[+0.006, +0.074]better on 28/39 |
| Lateral gain (1 = same size as the car's motion)closer to 1 is better; the delta is the change in distance from 1 | 0.754[0.665, 0.840]median 0.789 | not separated, interval includes zero+0.065[−0.014, +0.133]better on 14/39 | better, 95% interval excludes zero−0.054[−0.090, −0.028]better on 35/39 |
| Lateral action rate, RMS (smoothness)m/s³, no better direction | 0.422[0.343, 0.511]median 0.366 | lower, 95% interval excludes zero; no better direction−0.242[−0.293, −0.187]lower on 38/39 | higher, 95% interval excludes zero; no better direction+0.010[+0.004, +0.017]lower on 12/39 |
| Lead of the lateral request over the car's motions, no better direction | 0.466[0.366, 0.577]median 0.425n = 34 | higher, 95% interval excludes zero; no better direction+0.065[+0.009, +0.132]lower on 8/33 | not separated, interval includes zero−0.013[−0.032, +0.012]lower on 17/32 |
| Longitudinal errorm/s², lower is better | 0.302[0.260, 0.349]median 0.271 | not separated, interval includes zero−0.006[−0.020, +0.008]better on 20/39 | better, 95% interval excludes zero−0.015[−0.022, −0.009]better on 32/39 |
| Longitudinal correlationhigher is better | 0.729[0.630, 0.817]median 0.837 | not separated, interval includes zero+0.012[−0.008, +0.031]better on 21/39 | better, 95% interval excludes zero+0.012[+0.003, +0.024]better on 24/39 |
| Longitudinal error while brakingm/s², lower is better | 0.408[0.348, 0.477]median 0.388n = 30 | not separated, interval includes zero−0.027[−0.058, +0.010]better on 15/30 | better, 95% interval excludes zero−0.033[−0.054, −0.016]better on 25/30 |
| Longitudinal action rate, RMS (smoothness)m/s³, no better direction | 0.880[0.799, 0.971]median 0.816 | lower, 95% interval excludes zero; no better direction−0.453[−0.595, −0.321]lower on 37/39 | higher, 95% interval excludes zero; no better direction+0.113[+0.086, +0.139]lower on 3/39 |
| Plan speed error at 3 sm/s, lower is better | 0.758[0.658, 0.888]median 0.744 | not separated, interval includes zero+0.043[−0.010, +0.094]better on 14/39 | not separated, interval includes zeroidenticalon all 39 |
comma's public drives, closed loop
Not run for this model yet.
G55, open loop
4 windows from 3 drives.
Measured. The G55's recorded drives replayed as the truck runs the model: narrow camera only, road-plane warp to 1.45 m (plane:1.45). Truth is the truck's CAN speed and yaw rate. Window = unit. 4 windows: 3 on an earlier camera mount, 1 on the current mount; 2 at night.
| Metric | Mean | vs lebowski | vs Cinque v2 |
|---|---|---|---|
| Lateral errorm/s², lower is better | 0.147no intervalmedian 0.161 | no interval (too few drives)+0.020no intervalbetter on 0/40/3 drives | no interval (too few drives)−0.004no intervalbetter on 4/43/3 drives |
| Lateral error at the model's own best leadm/s², lower is better | 0.142no intervalmedian 0.153 | no interval (too few drives)+0.015no intervalbetter on 1/41/3 drives | no interval (too few drives)−0.001no intervalbetter on 1/41/3 drives |
| Lateral correlation with the car's later motionhigher is better | 0.946no intervalmedian 0.946 | no interval (too few drives)−0.015no intervalbetter on 1/41/3 drives | no interval (too few drives)+0.016no intervalbetter on 4/43/3 drives |
| Lateral gain (1 = same size as the car's motion)closer to 1 is better; the delta is the change in distance from 1 | 0.848no intervalmedian 0.846 | no interval (too few drives)+0.095no intervalbetter on 0/40/3 drives | no interval (too few drives)−0.023no intervalbetter on 2/41/3 drives |
| Lateral action rate, RMS (smoothness)m/s³, no better direction | 0.512no intervalmedian 0.510 | no interval (too few drives)−0.445no intervallower on 4/43/3 drives | no interval (too few drives)−0.006no intervallower on 3/42/3 drives |
| Lead of the lateral request over the car's motions, no better direction | 0.425no intervalmedian 0.475 | no interval (too few drives)+0.087no intervallower on 1/41/3 drives | no interval (too few drives)−0.037no intervallower on 2/41/3 drives |
| Longitudinal errorm/s², lower is better | 0.317no intervalmedian 0.317 | no interval (too few drives)+0.003no intervalbetter on 2/41/3 drives | no interval (too few drives)+0.002no intervalbetter on 1/41/3 drives |
| Longitudinal correlationhigher is better | 0.563no intervalmedian 0.620 | no interval (too few drives)−0.019no intervalbetter on 0/40/3 drives | no interval (too few drives)+0.001no intervalbetter on 3/42/3 drives |
| Longitudinal error while brakingm/s², lower is better | 0.584no intervalmedian 0.600 | no interval (too few drives)+0.069no intervalbetter on 1/41/3 drives | no interval (too few drives)−0.027no intervalbetter on 4/43/3 drives |
| Longitudinal action rate, RMS (smoothness)m/s³, no better direction | 1.27no intervalmedian 1.22 | no interval (too few drives)−0.48no intervallower on 4/43/3 drives | no interval (too few drives)+0.19no intervallower on 0/40/3 drives |
| Plan speed error at 3 sm/s, lower is better | 1.05no intervalmedian 1.05 | no interval (too few drives)+0.02no intervalbetter on 2/42/3 drives | not separated, interval includes zeroidenticalon all 4 |
G55, closed loop
Not run for this model yet.
G55, recovery tests
Not run for this model yet.
Limits
- Open-loop scores compare a model's action with what the recording driver or car did; a model that drives differently but well is scored as wrong. comma's RL residual makes the lateral action smoother, and earlier than its base's by 0.10 to 0.26 s (measured on the G55 and on 16 of comma's cars); both score as error here. Measured on the 2026-10-27 deep dive: open-loop accuracy did not predict closed-loop driving.
- Closed-loop numbers come from a generative world model, not a car. In a separate test on the G55's windows, worldmodel-4B followed an added yaw command through 2 s but showed 0 to 20% of an added lateral offset 3 to 4 s later; past about 4 s the recorded anchor frames also pull every model back toward the recorded path. The front-page ranking uses 2 s.
- Intervals are 95% percentile bootstraps that resample whole drives (the drive a public segment was cut from, the recorded drive a truck window comes from), so segments of one drive count once. Below 10 drives there is no interval: the truck sections come from fewer drives and report means and win counts only.
- Paired deltas are model minus baseline on the units both have (for lateral gain, the change in its distance from 1); wins count units where the model is better in the metric's direction.
- The lead of the lateral request is the shift, searched from 0 to 1.5 s, at which the request best correlates with the car's later lateral motion; a larger value means the request comes earlier. A value at either edge is censored and left out. Plan speed error at 3 s reads the plan head, which does not separate models built on one frozen base: it is identical on every segment for the four f22e06e5 models, and for Cinque v2 and v3.
- Truck open loop pools few windows (camera mounts changed between drives); read it as a check that a model works on the truck's camera, not a ranking.
The model file
- Model id
f78ed37d_12864- SHA-256
404a18cfd86d29637d20c697dfde245bb47c666ae016730ab674c65f4d1e1aa4- File scored
- huggingface,
81a9ad0139976653c041de50b60f1160a873869e(link), 2026-09-18 - First public
- 2026-09-18
- Parameters
- 381.6M
- GMAC per step
- 46.7
- Family
- big-rl
- Scored
- 2026-10-27 21:27 UTC, suite
geval-1, code102710f59c - Notes
- On openpilot master 2026-09-18 (PR #38932). Vision base b9facbcc, no residual head.
Results file (results/f78ed37d_12864.json, schema 1).