Feature request: Conversion from float retaining excess precision

[hniksic]

Currently this code prints 0.1:

let d = Decimal::from_f64(0.1).unwrap();
println!("{}", d);

At first that sounds like correct behavior and what one would want. But I would argue that it is in fact not the correct result, and that it should instead output 0.10000000000000000555111512313.

As we know, the token 0.1 is read by Rust into an f64 number, where it is represented by a floating-point number equal to the fraction 3602879701896397/36028797018963968. (This ratio can be obtained in Python using 0.1 .as_integer_ratio(), or in Rust by printing BigRational::from_float(0.1).) While the resulting number can be displayed as 0.1 without loss of precision when again re-read as float, and while humans often think of it as 0.1, it is the fraction that is in fact used in floating-point calculations.

This is why I would expect "0.1".parse::<Decimal>().unwrap() to differ from Decimal::from_f64(0.1).unwrap(). The former reads from decimal and is precisely 0.1, whereas the latter converts the result of the lossy conversion of the 0.1 token to floating point. The input of Decimal::from_f64(0.1) is the number corresponding to the above-mentioned fraction, which has an exact decimal representation (because its denominator is a power of 2): 0.1000000000000000055511151231257827021181583404541015625. Given the constraint of a mantissa smaller than 2**96, the closest rust_decimal::Decimal representation of this is 10000000000000000555111512313 * 10**-29, so I'd expect the Decimal to display as 0.10000000000000000555111512313.

The same applies to from_f32 where Decimal has enough precision to hold all the digits of 13421773/134217728, and should display 0.100000001490116119384765625.

Python's decimal does the equivalent conversion, except its decimal is not limited to u128, so it can accommodate the whole float:

>>> import decimal
>>> decimal.Decimal(0.1)
Decimal('0.1000000000000000055511151231257827021181583404541015625')
>>> decimal.Decimal("0.1")
Decimal('0.1')

Looking at the implementation, it's not at all obvious that the current behavior is intended. The intention seems to be to convert the number with maximum possible precision. While I'm aware that some people might consider the current behavior to be a feature, I'd still ask to change it, or at least to provide a different method for converting floats to decimals while minimizing the difference from the original number.

[paupino]

The code that is doing the "rounding" per se is this:

rust-decimal/src/decimal.rs

Lines 1498 to 1516 in 980c987

	if is64 {
	// Guaranteed to about 16 dp
	while exponent10 < 0 && (bits[2] != 0 || (bits[1] & 0xFFF0_0000) != 0) {
	let rem10 = ops::array::div_by_u32(bits, 10);
	exponent10 += 1;
	if rem10 >= 5 {
	ops::array::add_one_internal(bits);
	}
	}
	} else {
	// Guaranteed to about 7 dp
	while exponent10 < 0 && ((bits[0] & 0xFF00_0000) != 0 || bits[1] != 0 || bits[2] != 0) {
	let rem10 = ops::array::div_by_u32(bits, 10);
	exponent10 += 1;
	if rem10 >= 5 {
	ops::array::add_one_internal(bits);
	}
	}
	}

As you can see in this, it's removing excess precision after approx 16 decimal places for a f64 due to the fact that a 64 bit float can't guarantee more than that (as per IEEE-754). It then continues on to remove any trailing zeros here:

rust-decimal/src/decimal.rs

Lines 1518 to 1530 in 980c987

	// Remove multiples of 10 from the representation
	while exponent10 < 0 {
	let mut temp = [bits[0], bits[1], bits[2]];
	let remainder = ops::array::div_by_u32(&mut temp, 10);
	if remainder == 0 {
	exponent10 += 1;
	bits[0] = temp[0];
	bits[1] = temp[1];
	bits[2] = temp[2];
	} else {
	break;
	}
	}

Ultimately this is why it becomes 0.1 in the end - namely as that is what can be considered the approximate represented precision based upon IEEE-754 standards.

I think what you're asking for is really to have a function which doesn't remove this excess precision so that we ignore any float guarantees. My question I guess is: why? It'd be useful to understand the use case.

Btw, the way this handles floats is very similar to the .NET approach and gives identical solutions in this case.

[hniksic]

I think what you're asking for is really to have a function which doesn't remove this excess precision so that we ignore any float guarantees. My question I guess is: why? It'd be useful to understand the use case.

The use case is being able to closely approximate the number that f64 works with, for various reasons: rounding, inspection, education. At least I found Python's as_integer_ratio() to be extremely useful in various circumstances involving transitioning from floats to exact arithmetic in order to perform well-defined rounding operations for the purposes of display. By inspection and education I mean that exact conversions between floats and rationals allow one to model and deal with what otherwise seem like intractable phenomena like 0.1 + 0.1 + 0.1 - 0.3 evaluating to 5.551115123125783e-17.

"Ignoring any float guarantees" sounds almost like we'd be doing something unsafe, which I believe is not the case here, unless there is reason to doubt the soundness of the algorithm Python (and num::BigRational) uses to arrive at the fraction.

Having said that, I must also say that I didn't know about the guarantees you're referring to, and it's definitely interesting that .NET gives identical results. Do you think Python is just mistaken about their float->Decimal conversion? The documentation of decimal claims that its floating point arithmetic is "implemented in terms of published standards", including the hairy parts such as rounding and signal handiling. The documentation says that "Construction from an integer or a float performs an exact conversion of the value of that integer or float", giving examples like:

>>> Decimal('3.14')
Decimal('3.14')
>>> Decimal(3.14)
Decimal('3.140000000000000124344978758017532527446746826171875')

Decimal.from_float docs make a similar point explicitly:

Note Decimal.from_float(0.1) is not the same as Decimal(‘0.1’). Since 0.1 is not exactly representable in binary floating point, the value is stored as the nearest representable value which is 0x1.999999999999ap-4. That equivalent value in decimal is 0.1000000000000000055511151231257827021181583404541015625.

I understand that Python documentation is in no way normative, I'm quoting it merely to illustrate how far they go in considering the rational number to be some form of intrinsic value of the float. I would find it puzzling that they'd miss something as basic as a fundamental guarantee afforded by IEEE 754, or that they'd recklessly flout it.

Also, does the same apply to BigRational::from_float() - should it return 1/10 when given 0.1f64 and 0.1f32? I'm genuinely curious, not (only) trying to make a point. Maybe there are two different modes of float->decimal conversion, and both should be considered.

[paupino]

I think the primary difference between the libraries is that one of the design goals of Rust Decimal, rightly or wrongly, is to follow IEEE-754 as much as possible - at least where it makes sense.

I mean, ideally it's not my place to really say what's right or wrong here. It's much easier to provide customization so that you can extract the behavior that you need from the library - especially one so simple as this. Subsequently, I think there are probably two things that need to be addressed here:

1. The documentation is seriously lacking and needs fleshing out. I think it's worthwhile explaining what's happening here. It's also worth elaborating IEEE-754 and where we choose to follow it a bit more since it's not the first time we've needed to explain behavior.
2. Provide a second function to allow parsing while keeping the excess precision. This could be implemented easily without performance overhead (e.g. inline a private function accepting a bool - which will get optimized out). The only hard part would be figuring out what to name the function.

At a stretch this could also be enabled/disabled with a feature flag - though I'm a little hesitant to do it that way unless absolutely required.

[ahrvoje]

I tend toward Python and BOOST approach, for both consistency and practical reasons.

// .NET 5
new Decimal(1.1000000000000001) == Decimal.Parse("1.1")); // TRUE
new Decimal(1.1000000000000001) == Decimal.Parse("1.1000000000000001")); // FALSE

[schungx]

I think we are talking about philosophical differences here.

It is clear that 64 bits is not enough to represent all possible floating-point numbers. Therefore, for an f64, there are only 15-17 significant digits. Thus, anything after this number of significant digits is uncertain and fluffy.

Therefore, taking the only guaranteed number of digits is JUST AS CORRECT as making up a random stream of digits after that (of course those that do not round up) -- as far as IEEE floats are concerned, the representation scheme cannot distinguish between them. In other words, they all map to the exact same representation.

So, technically speaking, 1.1 and 1.10000000000000001 both map to the same IEEE number, and they are equally correct. The first decimal representation merely assumes all following digits to be zeros, while the second representation assumes the following digits just happen to follow the bit-patterns of the IEEE float. Technically, NONE of them are correct, because the real number is unknown.

So it is a matter of taste which representation to choose:

1. Take all zeros after max guaranteed digit --> 1.1
2. Just use the IEEE bit pattern --> 1.10000000000000001 etc.

But, you may also ask, why using the raw IEEE bit patterns would be useful or in any way more meaningful for any purpose? Mind as well zero the non-guaranteed digits - at least then the numbers are simpler; they both have similar distributions - the IEEE representation is deterministic.

[hniksic]

Thus, anything after this number of significant digits is uncertain and fluffy.

f64 represents a binary float in the form x*2**e (with constraints on the size of x and e). In case of 0.1, it is well-defined which x and e result in a number closest to 0.1 - those forming the expression 3602879701896397*2**-55 (0x1.999999999999ap-4 in floating-point binary). That number is perfectly representable in decimal as 0.1000000000000000055511151231257827021181583404541015625 - there's nothing "uncertain" or "fluffy" about the resulting decimal digits.

The decimal digits are an exact representation of the binary number stored in f64. This is why Python uses them, and so does Java, C++ Boost (when widening from e.g. double to boost::multiprecision::cpp_dec_float_100), Rust's BigRational, and probably many others.

But, you may also ask, why using the raw IEEE bit patterns would be useful or in any way more meaningful for any purpose?

The purpose is not to lose informattion. When converting from a narrower representation to a wider one, such as from f32 to f64, or from either of those to arbitrary-width decimal, no data should be lost (or loss should be minimized when converting to finite types such as rust_decimal::Decimal). For example, 0.1f32 as f64 doesn't evaluate to 0.1f64. But once you convert 0.1f32 to rust_decimal::Decimal with the current behavior, you would have no way to extract the original number that was in f32, so converting the resulting Decimal to f64 would result in 0.1f64, which is different to the original value stored in f32, as produced with 0.1f32 as f64.

let n1 = 0.1f32;
let n2 = 0.1f64;
let d1 = Decimal::from_f32(n1).unwrap();
let d2 = Decimal::from_f64(n2).unwrap();
println!("{} {} {}", d1, d2, n1 as f64 == n2);  // 0.1 0.1 false

With the proposed change, the output would be 0.100000001490116119384765625 0.10000000000000000555111512313 false, i.e. decimal conversions of the numbers would reflect the difference in their f32/f64 values.

[schungx]

But once you convert 0.1f32 to rust_decimal::Decimal with the current behavior, you would have no way to extract the original number that was in f32

I don't understand. Any number representation staying with the given number of significant digits will round-trip for IEEE floats.

Therefore, f32 -> Decimal -> f32 should always round-trip. Similarly for f64.

However, nothing is there to guarantee f32 -> Decimal -> f64 should be the same as f32 -> f64. This is what you're asking for, but the problem is not that you cannot get back the original bits pattern for the f32 (you can). The problem is using Decimal as a translator between two digital formats.

If you really want such conversion, you can always do:

f32 -> Decimal -> f32 (which always round-trips) -> f64 (which gets you your conversion)

[hniksic]

However, nothing is there to guarantee f32 -> Decimal -> f64 should be the same as f32 -> f64.

That's currently the case, but it doesn't have to be so, and it's not so with libraries that don't discard the full information from the float. And that's just one example (which you requested) where the loss of data by the current float->decimal conversion is easily observable.

[ahrvoje]

@schungx Yes, both Decimal values are different from double 1.1000000000000001, and this is exactly what you get in Python.

Python 3.10

Decimal(1.1000000000000001) == Decimal("1.1") # False
Decimal(1.1000000000000001) == Decimal("1.1000000000000001") # False

[schungx]

The number 1.1000000000000001 cannot be represented exactly by IEEE 754 double.

The closest machine number is 1.100000000000000088817841970012523233890533447265625.

Python simply ignores the fact that 1.1000000000000001 in float is fiction and silently gives you whatever the closest IEEE number is.

This crate, however, takes the opposite view that says anything after the 16th digit is not exact, and you most probably want to only know about at most 16 digits. Because... why do you use Decimal? Because you want exactness, that's why. To this end, why accept inexactness in the first place?

Personally, I take the view that an IEEE 754 number is not exact by design. Therefore, it is never supposed to be treated exactly. Simply taking whatever bit-pattern of that number and then taking it as gospel is at best misleading, and at worst wrong; a misuse.

Anything after the 16th digit in an IEEE 754 double is never supposed to be treated exactly and there is absolutely nothing wrong, even more intuitive, to just zero them off.

I'd say most programmers would expect something like:

let number: f64 = 1.1;
let decimal = Decimal::from(number);

Most people would intuitively expect the Decimal to be 1.1 instead of 1.100000000000000088817841970012523233890533447265625 which is the true IEEE 754 machine number for 1.1_f64.

And if you output 1.1_f64 as text, Rust will print 1.1 instead of 1.100000000000000088817841970012523233890533447265625 which is the exact IEEE 754 machine number. Again, I think I'd scream if it prints 1.100000000000000088817841970012523233890533447265625, forcing me to always round it before printing.

Because when I write 1.1, I mean 1.1, even though I know there will be more digits after 15 zeros, but I don't want them.

[hniksic]

Because... why do you use Decimal? Because you want exactness, that's why. To this end, why accept inexactness in the first place?

For me this is the argument to give me the exact number, with all its unsightly digits. There's nothing inexact about those digits, they are mathematically derived from the definition of the float. If there is inexactness, it happened much earlier, such as when the token 0.1 was converted to binary floating-point, but that's not under the control of this (or any other) crate.

Most people would intuitively expect the Decimal to be 1.1 instead of 1.100000000000000088817841970012523233890533447265625

Indeed they would, but the same people would probably also expect 0.1 + 0.2 - 0.3 to evaluate to 0 and not to 5.551115123125783e-17. Hiding the digits of the number that is actually stored makes sense for display purposes, especially if the display representation is tied to a particular type, so the extra digits don't convey additional information in the context of that type. But here we're discussing conversion to a wider type, one specifically designed to avoid pitfalls of binary floating point. In that case discarding information is not doing me any favors, and I'd expect there at least to be a way to convert the actual value stored.

If Python is just wrong here, then so is Java's BigDecimal, Rust's num::BigRational, and boost's cpp_dec_float_*, as well as floating-point casts like 0.1f32 as f64. I can understand that you don't need the actual precision in your use cases, but that doesn't mean that it has no place.

[paupino]

Would a feature flag to support this requirement work? This is a work in progress PR but could perhaps solve your requirements.

#441

With the feature flag float-excess-precision enabled the tests it_converts_from_f32 and it_converts_from_f64 will fail on the first number (0.1 since that is the example that is used in this thread):

---- it_converts_from_f32 stdout ----
thread 'it_converts_from_f32' panicked at 'assertion failed: `(left == right)`
  left: `"0.1"`,
 right: `"0.100000001490116119384765625"`: from_f32(0.1)', tests/decimal_tests.rs:2728:9

---- it_converts_from_f64 stdout ----
thread 'it_converts_from_f64' panicked at 'assertion failed: `(left == right)`
  left: `"0.1"`,
 right: `"0.1000000000000000055511151231"`: from_f64(0.1)', tests/decimal_tests.rs:2777:9

[hniksic]

Thanks for working on this!

Support through a feature is certainly better than no support at all, but features have several downsides, e.g. that they are hard to discover and document, that they are unavailable in some settings (famously the Playground, but there are other examples). Especially problematic are features that change behavior of existing functionality rather than just introduce new functions. Those interact badly with workspace builds, where, as I understand it, a crate is build as a union of the dependencies requested by various dependent crates in the same workspace. (I can dig out the exact details if you're curious.)

A pair of methods like from_f32_exact() and from_f64_exact() seem like a more versatile choice. Those could be clearly documented, and a user could choose the behavior based on their needs. If you don't like expanding the API surface, perhaps make them available via a FromFloatExact extension trait?

[paupino]

I tend to agree about the discoverability of feature flags vs introducing a new function. I've updated the PR however have gone for the names: from_f32_retain_excess_bits and from_f64_retain_excess_bits - a bit of a mouthful, however I think it's a more accurate representation of what is happening over from_f32_exact/from_f64_exact (i.e. it could be deemed that 0.1_f32 is exact when it's 1 x 10^-1 in Rust Decimal).

I could be convinced to shorten these to from_f32_retain/from_f64_retain. Anyway, the PR is still draft - just playing around with what it could be at the moment.

[ahrvoje]

One simple real-life example, as this is not philosophical discussion, it has real-life consequences and errors.

Which of two expressions y=1 - x*x and z=(1-x) * (1+x) is more accurate at x=1 - 1e-12?

Python 3.10

from decimal import Decimal

x = 1 - 1e-12

y = 1 - x*x
z = (1 - x) * (1 + x)

exact_x = Decimal(x)
exact_y = 1 - exact_x*exact_x
exact_z = (1 - exact_x) * (1 + exact_x)

print(abs(y - float(exact_y)))
print(abs(z - float(exact_z)))  # this error is smaller and this is CORRECT

Python 3.10 - 25 decimal digits precision

import decimal

decimal.getcontext().prec = 25

x = 1 - 1e-12

y = 1 - x*x
z = (1 - x) * (1 + x)

exact_x = decimal.getcontext().create_decimal(x)
exact_y = 1 - exact_x*exact_x
exact_z = (1 - exact_x) * (1 + exact_x)

print(abs(y - float(exact_y)))
print(abs(z - float(exact_z)))  # this error is smaller and this is CORRECT

.NET 5

using System;
					
public class Program
{
	public static void Main()
	{
		double x = 1 - 1e-12;

		double y = 1 - x*x;
		double z = (1 - x) * (1 + x);

		System.Decimal exact_x = new Decimal(x);
		System.Decimal exact_y = 1 - exact_x*exact_x;
		System.Decimal exact_z = (1 - exact_x) * (1 + exact_x);

		Console.WriteLine("error y = {0}", Math.Abs(y - Decimal.ToDouble(exact_y)));  // this error is smaller and this is INCORRECT
		Console.WriteLine("error z = {0}", Math.Abs(z - Decimal.ToDouble(exact_z)));
	}
}

Philosophy and personal preferences aside, this Decimal will not be useful for my purposes as I can not trust its results. Python Decimal gives correct answer even when crippled to 25 decimal digits precision, which is significantly below .NET Decimal precision, as it does not lose precision during conversion.

Its that simple.

[paupino]

To be clear, I'm not looking at changing the default behavior of this library and will continue to align with the .NET library in this regard. Because we are converting from a "base 2" representation into a "base 10" representation the additional precision may not be entirely accurate to what the "true" float represents either. Consequently, continuing to use a narrowing conversion based upon the calculated IEEE-754 decimal digits seems quite reasonable to cater for this inaccuracy.

That being said: perhaps there are alternative scenarios that you need a close approximation of the exp2 representation of the float. Given this, would the functions in #441 provide the additional functionality required? Are the results what you'd expect?

[hniksic]

from_f32_retain and from_f64_retain would be great names because they evoke that existing data is being retained without going into either the extreme of labeling it "exact" (which could indeed be confused with the exact decimal originally used to construct the float) or "excess" (which can be confused with e.g. excess bits coming from the CPU doing calculations in higher precision).

I'll check the results and report my findings in a comment to #441.

[hniksic]

@paupino Would you please consider publishing a release of the crate? This feature, together with the fixes for #437 and #430 provide ample motivation to have the latest code more readily available.