N16 nav model 96244b1f
2026-10-15 Hugging Face upload only, not shipped
Lateral action rate 29% lower and lateral error 40% higher than lebowski (39 segments). Against MD v3 / Flip Flop: higher lateral lead, lateral error and longitudinal error. Navigation inputs fed zeros.
Key numbers
The tick is lebowski. Lateral action rate and lateral lead have no better direction; on the others lower is better. Compared with lebowski and with MD v3 / Flip Flop 1365126a, the model first public before it.
All scores
better, 95% interval excludes zero Better, 95% interval excludes zeroworse, 95% interval excludes zero Worse, interval excludes zerolower, 95% interval excludes zero; no better directionhigher, 95% interval excludes zero; no better direction Lower or higher, no better direction (action rate, lead)not separated, interval includes zero Not separatedno interval (too few drives) No interval: fewer than 10 drives
comma's public drives, open loop
39 segments from 23 drives. Per-segment CSV.
Measured. comma's public two-camera segments (openpilot CI data), native intrinsics and the device's own calibration, both cameras; every model on the same decoded frames. Truth is what the recording car did next: carState speed and acceleration, yaw rate from the device's cameraOdometry. Segments where the car never moves have no data. 39 of the 45 listed segments have a score, from 23 drives and 20 cars.
| Metric | Mean | vs lebowski | vs MD v3 / Flip Flop |
|---|---|---|---|
| Lateral errorm/s², lower is better | 0.152[0.119, 0.185]median 0.134 | worse, 95% interval excludes zero+0.043[+0.029, +0.056]better on 3/39 | worse, 95% interval excludes zero+0.023[+0.013, +0.033]better on 5/39 |
| Lateral error at the model's own best leadm/s², lower is better | 0.142[0.110, 0.175]median 0.130 | worse, 95% interval excludes zero+0.040[+0.025, +0.056]better on 8/39 | worse, 95% interval excludes zero+0.022[+0.012, +0.033]better on 6/39 |
| Lateral correlation with the car's later motionhigher is better | 0.828[0.746, 0.897]median 0.920 | not separated, interval includes zero−0.030[−0.072, +0.005]better on 9/39 | worse, 95% interval excludes zero−0.019[−0.037, −0.004]better on 6/39 |
| Lateral gain (1 = same size as the car's motion)closer to 1 is better; the delta is the change in distance from 1 | 0.734[0.644, 0.827]median 0.721 | worse, 95% interval excludes zero+0.092[+0.010, +0.167]better on 11/39 | not separated, interval includes zero+0.028[−0.024, +0.076]better on 9/39 |
| Lateral action rate, RMS (smoothness)m/s³, no better direction | 0.471[0.380, 0.576]median 0.422 | lower, 95% interval excludes zero; no better direction−0.193[−0.235, −0.145]lower on 37/39 | not separated, interval includes zero+0.020[−0.008, +0.039]lower on 6/39 |
| Lead of the lateral request over the car's motions, no better direction | 0.539[0.429, 0.660]median 0.500n = 35 | higher, 95% interval excludes zero; no better direction+0.154[+0.085, +0.240]lower on 4/34 | higher, 95% interval excludes zero; no better direction+0.069[+0.034, +0.113]lower on 3/35 |
| Longitudinal errorm/s², lower is better | 0.317[0.274, 0.366]median 0.298 | not separated, interval includes zero+0.009[−0.008, +0.028]better on 14/39 | worse, 95% interval excludes zero+0.016[+0.004, +0.027]better on 14/39 |
| Longitudinal correlationhigher is better | 0.729[0.626, 0.817]median 0.847 | not separated, interval includes zero+0.011[−0.015, +0.036]better on 23/39 | not separated, interval includes zero−0.010[−0.021, +0.004]better on 13/39 |
| Longitudinal error while brakingm/s², lower is better | 0.436[0.373, 0.509]median 0.429n = 30 | not separated, interval includes zero+0.002[−0.025, +0.035]better on 12/30 | worse, 95% interval excludes zero+0.036[+0.013, +0.062]better on 7/30 |
| Longitudinal action rate, RMS (smoothness)m/s³, no better direction | 0.900[0.797, 1.014]median 0.821 | lower, 95% interval excludes zero; no better direction−0.432[−0.561, −0.322]lower on 36/39 | not separated, interval includes zero+0.011[−0.025, +0.043]lower on 13/39 |
| Plan speed error at 3 sm/s, lower is better | 0.767[0.656, 0.902]median 0.747 | worse, 95% interval excludes zero+0.051[+0.015, +0.085]better on 13/39 | not separated, interval includes zero+0.006[−0.012, +0.024]better on 18/39 |
comma's public drives, closed loop
Not run for this model yet.
G55, open loop
4 windows from 3 drives.
Measured. The G55's recorded drives replayed as the truck runs the model: narrow camera only, road-plane warp to 1.45 m (plane:1.45). Truth is the truck's CAN speed and yaw rate. Window = unit. 4 windows: 3 on an earlier camera mount, 1 on the current mount; 2 at night.
| Metric | Mean | vs lebowski | vs MD v3 / Flip Flop |
|---|---|---|---|
| Lateral errorm/s², lower is better | 0.165no intervalmedian 0.179 | no interval (too few drives)+0.037no intervalbetter on 0/40/3 drives | no interval (too few drives)+0.023no intervalbetter on 0/40/3 drives |
| Lateral error at the model's own best leadm/s², lower is better | 0.150no intervalmedian 0.156 | no interval (too few drives)+0.023no intervalbetter on 0/40/3 drives | no interval (too few drives)+0.017no intervalbetter on 0/40/3 drives |
| Lateral correlation with the car's later motionhigher is better | 0.927no intervalmedian 0.934 | no interval (too few drives)−0.033no intervalbetter on 0/40/3 drives | no interval (too few drives)−0.014no intervalbetter on 0/40/3 drives |
| Lateral gain (1 = same size as the car's motion)closer to 1 is better; the delta is the change in distance from 1 | 0.813no intervalmedian 0.812 | no interval (too few drives)+0.129no intervalbetter on 0/40/3 drives | no interval (too few drives)+0.016no intervalbetter on 0/40/3 drives |
| Lateral action rate, RMS (smoothness)m/s³, no better direction | 0.545no intervalmedian 0.545 | no interval (too few drives)−0.411no intervallower on 4/43/3 drives | no interval (too few drives)+0.001no intervallower on 2/42/3 drives |
| Lead of the lateral request over the car's motions, no better direction | 0.562no intervalmedian 0.650 | no interval (too few drives)+0.225no intervallower on 0/40/3 drives | no interval (too few drives)+0.087no intervallower on 0/40/3 drives |
| Longitudinal errorm/s², lower is better | 0.339no intervalmedian 0.329 | no interval (too few drives)+0.025no intervalbetter on 0/40/3 drives | no interval (too few drives)+0.008no intervalbetter on 1/41/3 drives |
| Longitudinal correlationhigher is better | 0.540no intervalmedian 0.597 | no interval (too few drives)−0.042no intervalbetter on 0/40/3 drives | no interval (too few drives)−0.005no intervalbetter on 2/42/3 drives |
| Longitudinal error while brakingm/s², lower is better | 0.556no intervalmedian 0.573 | no interval (too few drives)+0.040no intervalbetter on 1/41/3 drives | no interval (too few drives)+0.016no intervalbetter on 3/42/3 drives |
| Longitudinal action rate, RMS (smoothness)m/s³, no better direction | 1.82no intervalmedian 1.48 | no interval (too few drives)+0.07no intervallower on 2/42/3 drives | no interval (too few drives)+0.09no intervallower on 1/41/3 drives |
| Plan speed error at 3 sm/s, lower is better | 1.57no intervalmedian 1.31 | no interval (too few drives)+0.54no intervalbetter on 1/41/3 drives | no interval (too few drives)+0.27no intervalbetter on 1/40/3 drives |
G55, closed loop
Not run for this model yet.
G55, recovery tests
Not run for this model yet.
Limits
- Open-loop scores compare a model's action with what the recording driver or car did; a model that drives differently but well is scored as wrong. comma's RL residual makes the lateral action smoother, and earlier than its base's by 0.10 to 0.26 s (measured on the G55 and on 16 of comma's cars); both score as error here. Measured on the 2026-10-27 deep dive: open-loop accuracy did not predict closed-loop driving.
- Closed-loop numbers come from a generative world model, not a car. In a separate test on the G55's windows, worldmodel-4B followed an added yaw command through 2 s but showed 0 to 20% of an added lateral offset 3 to 4 s later; past about 4 s the recorded anchor frames also pull every model back toward the recorded path. The front-page ranking uses 2 s.
- Intervals are 95% percentile bootstraps that resample whole drives (the drive a public segment was cut from, the recorded drive a truck window comes from), so segments of one drive count once. Below 10 drives there is no interval: the truck sections come from fewer drives and report means and win counts only.
- Paired deltas are model minus baseline on the units both have (for lateral gain, the change in its distance from 1); wins count units where the model is better in the metric's direction.
- The lead of the lateral request is the shift, searched from 0 to 1.5 s, at which the request best correlates with the car's later lateral motion; a larger value means the request comes earlier. A value at either edge is censored and left out. Plan speed error at 3 s reads the plan head, which does not separate models built on one frozen base: it is identical on every segment for the four f22e06e5 models, and for Cinque v2 and v3.
- Truck open loop pools few windows (camera mounts changed between drives); read it as a check that a model works on the truck's camera, not a ranking.
- This model takes a navigation map and route; every run here feeds zeros (no route), which is not how it would be used with navigation.
The model file
- Model id
96244b1f_12800- SHA-256
01effc679eb46d6036ce0302673a87fe2dcfcc95a679daeeb929679e53c3647a- File scored
- huggingface,
3fe6f3dea4fa9613dd3582b9a3828e97d307ad9e(link), 2026-10-15 - First public
- 2026-10-15
- Parameters
- 393.7M
- GMAC per step
- 51.9
- Family
- residual
- Scored
- 2026-10-27 21:27 UTC, suite
geval-1, code102710f59c - Notes
- Uploaded as "add N16 nav model"; never shipped. Takes a navigation map and route inputs, fed zeros here (no route). Vision base 2931d3d9 with an RL residual head.
Results file (results/96244b1f_12800.json, schema 1).