Unexpected return value

[gagmeng]

_2024_01_24_18_26_41_464.mp4

Numara does not support 64-bit data types; is that correct?

[bornova]

Hi @gagmeng,
Yes, I believe 64-bit data puts the answer out of range of the calculator so it may give unexpected results. However, converting the decimal value to a BigNumber first would fix the problem. You can try:

format(bignumber('-830'), {notation: "hex", wordSize: 64})  // 0xfffffffffffffcc2i64

Similar issue with the second part of your questions. Numbers greater than 15 significant digits would need to be converted to bignumber first. Here is an example:

hex(bignumber('18446744073709550') + 2931)  // 0x4189374bc6b361
hex(bignumber('18446744073709551') + 2931)  // 0x4189374bc6b362
hex(bignumber('18446744073709552') + 2931)  // 0x4189374bc6b363
hex(bignumber('18446744073709553') + 2931)  // 0x4189374bc6b364

On that note, I did find that using a number instead of string value in bignumber function gives incorrect result:

hex(bignumber(18446744073709551) + 2931)  // 0x4189374bc6b361
hex(bignumber('18446744073709551') + 2931)  // 0x4189374bc6b362

This may be a bug with Math.js and I will report it there.

Thank you!

[bornova]

On that note, I did find that using a number instead of string value in bignumber function gives incorrect result:

hex(bignumber(18446744073709551) + 2931)  // 0x4189374bc6b361
hex(bignumber('18446744073709551') + 2931)  // 0x4189374bc6b362

This may be a bug with Math.js and I will report it there.

I posted the question here and looks like it has to do with the rounding errors when it comes to javascript handling large numbers.

[gagmeng]

[bornova]

Hi @gagmeng,
Using bignumbers has its limitations unfortunately 😞 but I am not getting the same results on my end with default settings:

Have you changed any settings? and can you please the share the Error message.

[gagmeng]

The settings are below:

Best regards,
gagmeng

[bornova]

Hi @gagmeng,
Sorry I was away for a few days. Looks like you are the limitation of javascript when it comes to dealing with large numbers unfortunately. 😞 v4.4.3 does have a minor improvement to BigNumber but not sure if it will resolve your issues. Since I rely on Math.js, I wouldn't be able to offer much help on this.