Precision and rounding in conversions
Why two correct converters disagree in the fourth decimal place, and how many digits you are actually entitled to keep.
Unit conversion looks like the most deterministic thing a computer can do: multiply by a constant. In practice, three separate effects make converters disagree with each other and with printed tables: significant figures, binary floating point, and rounding applied more than once. None of them is a bug in the arithmetic, and all three are avoidable once you can tell them apart.
Precision is a property of the measurement, not the calculator
If someone tells you a room is 5 metres long, they are not claiming it is 5.000 000 metres. They are claiming it is closer to 5 than to 4 or 6. Converting that to 16.404 199 feet asserts a precision that was never in the original number: it implies you know the length to within a hundredth of a millimetre. The rule is that a converted value carries the same number of significant figures as the value it came from. Trailing digits beyond that are not extra accuracy; they are noise formatted to look like accuracy.
Binary floating point cannot represent most decimals
Every mainstream language stores non-integer numbers as IEEE 754 doubles, which are binary fractions. 0.1 has no exact binary representation, in the same way 1/3 has no exact decimal one. This is why 0.1 + 0.2 evaluates to 0.30000000000000004 in JavaScript, Python, Java and C. A double carries roughly 15 to 17 significant decimal digits, so results are reliable well past any real-world measurement, but if you print all the digits, you will eventually print the error. Format to the precision you actually have.
Rounding at every step compounds
Converting metres to feet to inches, rounding at each stage, is not the same as converting metres to inches once. Each rounding introduces up to half a unit of error in the last place, and chained conversions accumulate them in the same direction as often as not. Carry full precision through the whole calculation and round exactly once, at the point of display. This is also why a conversion table built by converting the previous row rather than the original value drifts as you go down it.
Exact conversions and measured ones are different things
1 inch = 25.4 mm is a definition, so the conversion is exact and you may keep as many digits as you like. 1 pound = 4.448 221 615 260 5 newtons of force is also exact, because it derives from a defined pound and a defined standard gravity. But 1 atmosphere = 14.7 psi is a rounded convention, and a nautical mile was historically a measured quantity before being fixed at exactly 1852 m. Knowing which kind you are holding tells you whether the trailing digits mean anything.
Significant figures in practice
The same conversion factor, applied to inputs that differ only in written precision, should produce answers that differ in written precision too.
| Input | Raw result | Report as | Why |
|---|---|---|---|
| 5 km | 3.106 855 miles | 3.1 miles | One significant figure in, one out |
| 5.0 km | 3.106 855 miles | 3.1 miles | Two significant figures |
| 5.00 km | 3.106 855 miles | 3.11 miles | Three significant figures |
| 25.4 mm | 1.000 000 inches | 1 inch exactly | Definitional, not measured |
A rule of thumb
- Carry every digit through the calculation; round once, when you display.
- Match the significant figures of the input, not the width of the display.
- Never build a table by converting the previous row.
- For money, use integer minor units or a decimal type, never a float.
- If the conversion is definitional, say so; the extra digits are real.
Try it yourself
Every calculation runs locally in your browser; nothing is sent anywhere.
Related: metric vs imperial lists which conversions are exact by definition, and conversion mistakes covers what happens when precision errors reach production.