Skip to content
PreciSim
Back to the blog
2 min read

A small residual and a wrong calibration

Everyone doing calibration watches the residual. A chain finishes, RMS is 0.08 px, and you relax.

That relief is often unearned.

Two different things

  • Residual: how well the model fits the data used in the solve;
  • Deviation: how far the result sits from the truth.

A small residual says exactly one thing: your model can explain the data you collected. If the data carries a systematic problem, the model absorbs that problem too — and hands you a pretty residual.

An example you can reproduce

On demo-2axis-vision:

  1. Run pixel size calibration normally: residual RMS 0.081 px, deviation 0.018 mm, PASS.
  2. Change lens distortion k1 from −0.0214 to −0.08 — the equivalent of fitting a cheaper lens and not re-calibrating distortion.
  3. Re-run acquire and solve only.
Residual RMS Deviation
Normal 0.081 px 0.018 mm
Distortion uncalibrated 0.213 px 0.150 mm

The residual grew 2.6× — that looks survivable. The deviation grew 8.3× and is three times over tolerance.

Watch only the residual and you conclude "a bit worse, still usable". In reality the machine can no longer do its job.

Why it happens

Distortion is a systematic offset that varies with radius. When solving for pixel size, least squares finds a compromise mm/px that keeps every point's residual moderate — pulling centre points out a little, edge points in a little. On average the residual still looks fine.

But that compromise is wrong at every actual position. On the line, parts near the centre of the field come out accurate and parts at the edge do not, which presents as "it works most of the time".

How to expose it

Re-project data that was held out of the solve. That is the entire purpose of the verify step in a calibration chain.

In practice:

  • collect N poses, solve on 70 %, hold back 30 %;
  • predict the held-out positions with the solved parameters and compare;
  • that error is the number to trust.

A virtual device goes one better: it knows the truth, so it can report deviation directly without holding anything back. Real hardware has no such luxury, which is why the held-out set is not optional there.

Three rules you can apply today

  1. Never solve and verify on the same data. It is the most common piece of self-deception in calibration reports.
  2. When a residual looks odd, suspect the data before you touch the model. Adding higher-order distortion terms will push the residual down — by fitting noise, while the deviation gets worse.
  3. Span beats pose count. Nine poses across 50 mm beat fifty poses across 5 mm. Noise averages down in proportion to span × √poses, and span is the linear term.

Further reading