6b5b705f
2026-10-17 Hugging Face upload only, not shipped
Lateral action rate 30% lower, lateral error 26% higher and longitudinal error 19% higher than lebowski (39 segments). Against N16 nav model: lower lateral error; higher longitudinal error.
Key numbers
The tick is lebowski. Lateral action rate and lateral lead have no better direction; on the others lower is better. Compared with lebowski and with N16 nav model 96244b1f, the model first public before it.
All scores
better, 95% interval excludes zero Better, 95% interval excludes zeroworse, 95% interval excludes zero Worse, interval excludes zerolower, 95% interval excludes zero; no better directionhigher, 95% interval excludes zero; no better direction Lower or higher, no better direction (action rate, lead)not separated, interval includes zero Not separatedno interval (too few drives) No interval: fewer than 10 drives
comma's public drives, open loop
39 segments from 23 drives. Per-segment CSV.
Measured. comma's public two-camera segments (openpilot CI data), native intrinsics and the device's own calibration, both cameras; every model on the same decoded frames. Truth is what the recording car did next: carState speed and acceleration, yaw rate from the device's cameraOdometry. Segments where the car never moves have no data. 39 of the 45 listed segments have a score, from 23 drives and 20 cars.
| Metric | Mean | vs lebowski | vs N16 nav model |
|---|---|---|---|
| Lateral errorm/s², lower is better | 0.137[0.108, 0.168]median 0.118 | worse, 95% interval excludes zero+0.028[+0.020, +0.037]better on 4/39 | better, 95% interval excludes zero−0.015[−0.023, −0.007]better on 25/39 |
| Lateral error at the model's own best leadm/s², lower is better | 0.126[0.098, 0.155]median 0.108 | worse, 95% interval excludes zero+0.024[+0.015, +0.033]better on 4/39 | better, 95% interval excludes zero−0.016[−0.026, −0.007]better on 25/39 |
| Lateral correlation with the car's later motionhigher is better | 0.837[0.748, 0.912]median 0.943 | not separated, interval includes zero−0.021[−0.066, +0.014]better on 10/39 | not separated, interval includes zero+0.009[−0.001, +0.020]better on 28/39 |
| Lateral gain (1 = same size as the car's motion)closer to 1 is better; the delta is the change in distance from 1 | 0.823[0.724, 0.920]median 0.879 | not separated, interval includes zero+0.028[−0.036, +0.081]better on 13/39 | better, 95% interval excludes zero−0.064[−0.097, −0.035]better on 31/39 |
| Lateral action rate, RMS (smoothness)m/s³, no better direction | 0.464[0.371, 0.571]median 0.410 | lower, 95% interval excludes zero; no better direction−0.200[−0.242, −0.155]lower on 38/39 | not separated, interval includes zero−0.007[−0.020, +0.008]lower on 25/39 |
| Lead of the lateral request over the car's motions, no better direction | 0.558[0.446, 0.680]median 0.475n = 36 | higher, 95% interval excludes zero; no better direction+0.169[+0.075, +0.278]lower on 4/35 | not separated, interval includes zero0.000[−0.049, +0.061]lower on 18/35 |
| Longitudinal errorm/s², lower is better | 0.368[0.321, 0.416]median 0.356 | worse, 95% interval excludes zero+0.060[+0.035, +0.086]better on 5/39 | worse, 95% interval excludes zero+0.051[+0.036, +0.067]better on 3/39 |
| Longitudinal correlationhigher is better | 0.674[0.550, 0.780]median 0.814 | worse, 95% interval excludes zero−0.043[−0.088, −0.003]better on 15/39 | worse, 95% interval excludes zero−0.054[−0.089, −0.026]better on 6/39 |
| Longitudinal error while brakingm/s², lower is better | 0.461[0.392, 0.538]median 0.444n = 30 | not separated, interval includes zero+0.026[−0.005, +0.064]better on 10/30 | not separated, interval includes zero+0.024[0.000, +0.048]better on 8/30 |
| Longitudinal action rate, RMS (smoothness)m/s³, no better direction | 1.01[0.88, 1.16]median 0.95 | lower, 95% interval excludes zero; no better direction−0.32[−0.45, −0.20]lower on 33/39 | higher, 95% interval excludes zero; no better direction+0.11[+0.06, +0.17]lower on 9/39 |
| Plan speed error at 3 sm/s, lower is better | 0.760[0.657, 0.888]median 0.712 | worse, 95% interval excludes zero+0.044[+0.009, +0.077]better on 12/39 | not separated, interval includes zero−0.006[−0.024, +0.012]better on 21/39 |
comma's public drives, closed loop
Not run for this model yet.
G55, open loop
4 windows from 3 drives.
Measured. The G55's recorded drives replayed as the truck runs the model: narrow camera only, road-plane warp to 1.45 m (plane:1.45). Truth is the truck's CAN speed and yaw rate. Window = unit. 4 windows: 3 on an earlier camera mount, 1 on the current mount; 2 at night.
| Metric | Mean | vs lebowski | vs N16 nav model |
|---|---|---|---|
| Lateral errorm/s², lower is better | 0.152no intervalmedian 0.164 | no interval (too few drives)+0.024no intervalbetter on 0/40/3 drives | no interval (too few drives)−0.013no intervalbetter on 4/43/3 drives |
| Lateral error at the model's own best leadm/s², lower is better | 0.142no intervalmedian 0.150 | no interval (too few drives)+0.016no intervalbetter on 0/40/3 drives | no interval (too few drives)−0.007no intervalbetter on 4/43/3 drives |
| Lateral correlation with the car's later motionhigher is better | 0.937no intervalmedian 0.941 | no interval (too few drives)−0.023no intervalbetter on 0/40/3 drives | no interval (too few drives)+0.011no intervalbetter on 4/43/3 drives |
| Lateral gain (1 = same size as the car's motion)closer to 1 is better; the delta is the change in distance from 1 | 0.865no intervalmedian 0.882 | no interval (too few drives)+0.077no intervalbetter on 0/40/3 drives | no interval (too few drives)−0.052no intervalbetter on 4/43/3 drives |
| Lateral action rate, RMS (smoothness)m/s³, no better direction | 0.556no intervalmedian 0.570 | no interval (too few drives)−0.401no intervallower on 4/43/3 drives | no interval (too few drives)+0.011no intervallower on 2/41/3 drives |
| Lead of the lateral request over the car's motions, no better direction | 0.475no intervalmedian 0.525 | no interval (too few drives)+0.138no intervallower on 0/40/3 drives | no interval (too few drives)−0.087no intervallower on 3/42/3 drives |
| Longitudinal errorm/s², lower is better | 0.372no intervalmedian 0.383 | no interval (too few drives)+0.057no intervalbetter on 0/40/3 drives | no interval (too few drives)+0.032no intervalbetter on 1/41/3 drives |
| Longitudinal correlationhigher is better | 0.513no intervalmedian 0.575 | no interval (too few drives)−0.069no intervalbetter on 0/40/3 drives | no interval (too few drives)−0.027no intervalbetter on 1/41/3 drives |
| Longitudinal error while brakingm/s², lower is better | 0.569no intervalmedian 0.639 | no interval (too few drives)+0.054no intervalbetter on 1/40/3 drives | no interval (too few drives)+0.014no intervalbetter on 1/41/3 drives |
| Longitudinal action rate, RMS (smoothness)m/s³, no better direction | 1.88no intervalmedian 1.23 | no interval (too few drives)+0.13no intervallower on 2/41/3 drives | no interval (too few drives)+0.06no intervallower on 1/41/3 drives |
| Plan speed error at 3 sm/s, lower is better | 1.30no intervalmedian 1.03 | no interval (too few drives)+0.28no intervalbetter on 1/41/3 drives | no interval (too few drives)−0.27no intervalbetter on 3/43/3 drives |
G55, closed loop
Not run for this model yet.
G55, recovery tests
Not run for this model yet.
Limits
- Open-loop scores compare a model's action with what the recording driver or car did; a model that drives differently but well is scored as wrong. comma's RL residual makes the lateral action smoother, and earlier than its base's by 0.10 to 0.26 s (measured on the G55 and on 16 of comma's cars); both score as error here. Measured on the 2026-10-27 deep dive: open-loop accuracy did not predict closed-loop driving.
- Closed-loop numbers come from a generative world model, not a car. In a separate test on the G55's windows, worldmodel-4B followed an added yaw command through 2 s but showed 0 to 20% of an added lateral offset 3 to 4 s later; past about 4 s the recorded anchor frames also pull every model back toward the recorded path. The front-page ranking uses 2 s.
- Intervals are 95% percentile bootstraps that resample whole drives (the drive a public segment was cut from, the recorded drive a truck window comes from), so segments of one drive count once. Below 10 drives there is no interval: the truck sections come from fewer drives and report means and win counts only.
- Paired deltas are model minus baseline on the units both have (for lateral gain, the change in its distance from 1); wins count units where the model is better in the metric's direction.
- The lead of the lateral request is the shift, searched from 0 to 1.5 s, at which the request best correlates with the car's later lateral motion; a larger value means the request comes earlier. A value at either edge is censored and left out. Plan speed error at 3 s reads the plan head, which does not separate models built on one frozen base: it is identical on every segment for the four f22e06e5 models, and for Cinque v2 and v3.
- Truck open loop pools few windows (camera mounts changed between drives); read it as a check that a model works on the truck's camera, not a ranking.
The model file
- Model id
6b5b705f_12800- SHA-256
ceb38c593005b3338aee4ca057d5ad9ee64c58f8fdba0b41641a9d4cc6c36e36- File scored
- huggingface,
eed9e515dccd5aa9be969a2d6c57eb469174a97d(link), 2026-10-17 - First public
- 2026-10-17
- Parameters
- 382.9M
- GMAC per step
- 50.1
- Family
- residual
- Scored
- 2026-10-27 21:27 UTC, suite
geval-1, code102710f59c - Notes
- Never shipped. Same frozen base as MD v3 (f22e06e5/56320, bit-identical), different RL residual head.
Results file (results/6b5b705f_12800.json, schema 1).