erts: Round bignum to float conversion correctly

[yaglo]

big_to_double/2 accumulated the result one digit at a time:

while (xl--) {
    d = d * dbase + *--s;
    if (!erts_isfinite(d)) return -1;
}

Every iteration rounds, so the errors compound and the result can be the second-nearest double rather than the nearest. IEEE 754 requires the nearest representable value, ties to even.

A value that fits in a single digit rounds only once and was already correct, so the defect starts at two digits. On a 64-bit build that means integers above 2^64:

1> float(428654966685883400000).
4.2865496668588343e20        %% second-nearest
2> binary_to_float(<<"428654966685883400000.0">>).
4.286549666858834e20         %% nearest

Both call sites describe the same real number, so they should agree. binary_to_float/1 routes through the platform's correctly rounded decimal parser and returns the nearest double; float/1 does not. Around 16% of 65-bit integers converted to the wrong double, and never to the nearer one — the error was systematic, not a tie-breaking difference.

Fix

Compute the bit length, and for values wider than the 53-bit mantissa take the top 54 bits (53 of mantissa plus one round bit) while recording whether any lower bit is set, then round exactly once. Values of 53 bits or fewer keep the digit-by-digit accumulation, which cannot round there.

Collecting the sticky bit visits every digit, so integers wider than 1024 bits return -1 up front: they are at least 2^1024 and cannot be finite. The previous loop stopped as soon as the accumulator went infinite, and without this check a conversion would become O(size) for bignums that reach tens of thousands of digits. Exactly 1024 bits can still be finite and takes the rounding path, where the existing overflow check after ldexp() catches a mantissa that carries up to 2^1024.

Converting (1 bsl 4000000) - 1 takes 24 us with the short-circuit and 1917 us without it, measured on the 32-bit build described below.

Testing

big_float_3 covers the reproducers, random values from 50 to 300 bits, and every power of two up to 2^1023. It fails on the previous implementation.

big_SUITE (23/23) and float_SUITE (15/15) pass on this branch.

The rounding boundary was checked explicitly, using the fact that the largest finite double is 2^1024 - 2^971:

input	result
trunc(max double)	1.7976931348623157e308
max + 2^970 (tie, rounds up)	overflow
max + 2^970 - 1	1.7976931348623157e308
2^1024 - 1 (1024 bits)	overflow
2^1023	8.98846567431158e307
2^1024, 2^1025, 2^2000	overflow

The tie case confirms that 1024-bit values still go through the rounding path rather than being rejected by the new short-circuit.

The digit-straddling arithmetic is the only part that depends on D_EXP, so it was checked two ways.

First, the algorithm was extracted into a standalone harness and instantiated at both 32-bit and 64-bit digit widths, then run over 94,116 vectors checked against a correctly rounded oracle: every bit width from 1 to 300, all powers of two to 2^1023 with their neighbours, and exact ties with their neighbours. Both widths were correct on every vector and agreed with each other bit for bit; 1,143 of those vectors are mis-rounded by the previous implementation.

Second, it was built and run on a 32-bit target (i686-pc-linux-gnu, wordsize 4, so D_EXP is 32). The reproducers all convert correctly; 220,000 random values spanning 53 to 500 bits and a further 300,000 80-bit values show no divergence from binary_to_float/1; negation stays symmetric over 50,000 values; and 1 bsl 2000 still raises badarith.

Provenance

The defect surfaced while implementing RFC 8785 (JSON Canonicalization Scheme). Canonical JSON numbers are IEEE 754 doubles, so a canonicalizer handed a large integer must decide whether it is exactly representable — convert it to the nearest double, and check whether the decimal digits come back unchanged.

That made the conversion itself worth testing. float/1 and binary_to_float/1 are given the same real number by construction, so they should return the same double; over random integers they disagreed often enough to be reproducible. Measuring which result was nearer the integer settled which one was wrong: it was never float/1, and there were no ties.

[CLAassistant]

All committers have signed the CLA.

[github-actions]

CT Test Results

    3 files    136 suites   51m 31s ⏱️
1 684 tests 1 628 ✅ 56 💤 0 ❌
2 327 runs  2 253 ✅ 74 💤 0 ❌

Results for commit c2a61ce.

♻️ This comment has been updated with latest results.

To speed up review, make sure that you have read Contributing to Erlang/OTP and that all checks pass.

See the TESTING and DEVELOPMENT HowTo guides for details about how to run test locally.

Artifacts

• Complete CT logs (Download Logs)
• HTML Documentation (Download HTML Docs)
• Windows Installer

// Erlang/OTP Github Action Bot