Inconsistent rounding of significant digits about d3-format

[mbostock]

The values you are using cannot be represented exactly by JavaScript numbers due to the nature of IEEE 754 floating point. More precise representations of the values you are using, as computed by number.toFixed(20), are:

• 1.55500 ↦ 1.55499999999999993783
• 2.55500 ↦ 2.55500000000000015987
• 3.55500 ↦ 3.55500000000000015987
• 4.55500 ↦ 4.55499999999999971578
• 5.55500 ↦ 5.55499999999999971578
• 6.55500 ↦ 6.55499999999999971578
• 7.55500 ↦ 7.55499999999999971578
• 8.55500 ↦ 8.55499999999999971578
• 9.55500 ↦ 9.55499999999999971578

As you can see, the two values that you consider to exhibit inconsistent behavior in D3 are the two values that are greater than the desired exact values, while the other values are all slightly less than the desired exact values.

For more on this topic, see http://0.30000000000000004.com/.

[drosen0]

But String(1.55499999999999993783) returns 1.555, as does toPrecision(16). JavaScript's String<->Number coercion understands that double precision floating point is only accurate to a precision of 16 digits.

Here's a modified formatDecimal.js that makes the results of r formatting consistent (with common rounding) up to .15r. If you'd consider implementing this sort of a fix, I'll happily submit a PR that also fixes e, f, and g.

// Computes the decimal coefficient and exponent of the specified number x with
// significant digits p, where x is positive and p is in [1, 21] or undefined.
// For example, formatDecimal(1.23) returns ["123", 0].
export default function(x, p) {
  if (x == null || isNaN(x) || x === Infinity || x === -Infinity) { return null; }

  var [mantissa, exponent='0'] = String(x).split('e');
  exponent = +exponent;
  var [whole, fractional=''] = mantissa.split('.');
  var digits = whole + fractional;

  digits = digits.replace(/^0+/, function (match) {
    exponent -= match.length;
    return '';
  });

  if (digits.length < p) {
    digits += Array(p - digits.length + 1).join('0');
  }

  digits = digits.slice(0, p) + '.' + digits.slice(p);
  digits = String(Math.round(digits));

  if (digits === '0') {
    digits = Array(p).join('0');
  }

  return [
    digits,
    exponent + whole.length - 1
  ];
}

[mbostock]

You don’t need to coerce to a string to see that JavaScript (IEEE 754) considers 1.555 and 1.55499999999999993783 to be equal. But that does not mean that mathematically the value is exactly 1.555. If you read the ECMAScript specification, you will see that JavaScript uses the minimum number of digits to uniquely identify a value, and so since these two values are the same from the context of floating point, the shorter representation is chosen. But this is arguably misleading in this context because it gives you a less accurate representation that using 20 digits. (Though naturally since IEEE 754 is binary and not decimal, 20 digits are not enough for an exact representation, either.)

So, your implementation effectively performs rounding twice, first by coercing to a string and then according to the desired behavior in D3. That means that you are treating the value 1.555 as exactly 1.555 (because this is the shortest unique decimal representation in IEEE 754) rather than its natural value of 1.55499999999999993783. Normally I would say this is bad in the context of mathematical operations, but perhaps it makes sense in the context of formatting values? It’s hard to say.

[mbostock]

Here’s further evidence that changing this behavior would be bad. Look at the native behavior of number.toPrecision:

(0.555).toPrecision(2) // "0.56" ⚠️
(1.555).toPrecision(3) // "1.55"
(2.555).toPrecision(3) // "2.56" ⚠️
(3.555).toPrecision(3) // "3.56" ⚠️
(4.555).toPrecision(3) // "4.55"
(5.555).toPrecision(3) // "5.55"
(6.555).toPrecision(3) // "6.55"
(7.555).toPrecision(3) // "7.55"
(8.555).toPrecision(3) // "8.55"
(9.555).toPrecision(3) // "9.55"

[mbostock]

This is related to d3/d3-path#10 (comment).

There are two valid approaches to decimal rounding. The first approach treats the input value x as its “natural” value in binary floating point. The second approach treats the input value x as the shortest-equivalent decimal value, as in the mathematical quantity represented by the result of calling number.toString.

The built-in methods number.toFixed and number.toPrecision chose the first approach, so the result of +(1.555).toFixed(2) is 1.55 because 1.555 is more precisely represented as 1.55499999999999993783 in binary floating point, which is less than 1.555, and thus rounds down.

If you instead interpret 1.555 as its shortest-equivalent decimal value, then it represents exactly 1.555, and thus should round up. And furthermore, it is possible to implement this technique in JavaScript, but it requires string concatentation: +(Math.round(1.555 + "e2") + "e-2") is 1.56. (See also round10.)

As @drosen0 stated, rounding should behave consistently. Applying decimal rounding rules to binary floating point values is perhaps surprising, but it is self-consistent (assuming there’s no implementation bug, which there does not appear to be), and furthermore is consistent with JavaScript’s built-in methods such as number.toFixed and number.toPrecision.

[pietersv]

Does any one know if there is a fork of d3.format that uses Math.round? I love the expressive range of d3.format syntax and its wide use in leading libraries, and get why it is the way it is now. One use of the library is to format numbers for human consumption, who would expect $8.995 to round to $9.99 either simply (ties-round-up) or scientific round-to-even. It's harder to explain that $8.995 is numerically the same as $8.9949999999999992 and thus should round down.