Not Quite "High Precision"

[serialhex]

Playing around in the interactive editor, I got a wrong answer that I kinda expected:

≫ 100000000000 + 0.1
  100000000000 + 0.1
   = 100000000000

Sure, I'm adding one tenth to 100 billion, and it's quite insignificant in compare, but I should be able to do this if I'm using a "high precision" calculator. I think the only way to fix this is to stop using IEEE floating point numbers, which is unfortunately the only number type in JavaScript.

[sharkdp]

Thank you for your feedback.

I think the only way to fix this is to stop using IEEE floating point numbers, which is unfortunately the only number type in JavaScript.

Insect does not use JavaScripts Number type (see "High-precision numeric type" in the README and "What is the internal numerical precision?" in the FAQs).

Playing around in the interactive editor, I got a wrong answer that I kinda expected:

≫ 100000000000 + 0.1
100000000000 + 0.1
= 100000000000

As stated in the README/FAQ, the internal result is accurate up to 30 digits, so you don't actually loose the 0.1 part:

>>> x = 100000000000 + 0.1

  x = 100000000000

>>> x - 99999999999

  x - 99999999999

   = 1.1

The problem is that Insect only shows 6 significant digits. This is also documented in the open issue #54 which reports the same problem. There is a plan to make this user-configurable (see #107).

Given that we use a numeric type with 30 digits, I hope you feel comfortable with me calling Insect a "high precision" numeric calculator.

[serialhex]

Huh, I guess I didn't read everything as thoroughly as I probably should have. Sorry about that.

I'll have to take a look at how you got the 30 digit numeric type... that kinda interests me!

[sharkdp]

Huh, I guess I didn't read everything as thoroughly as I probably should have. Sorry about that.

No worries! 😃

I'll have to take a look at how you got the 30 digit numeric type... that kinda interests me!

Insect uses this library: https://github.com/MikeMcl/decimal.js/