Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Calculation accuracies #332

Open
NicoLaval opened this issue May 20, 2024 · 2 comments
Open

Calculation accuracies #332

NicoLaval opened this issue May 20, 2024 · 2 comments

Comments

@NicoLaval
Copy link
Collaborator

@noahboerger reported:

  • In trevas the result of an operation is kept in double accuracy (e.g. numeric/power_1)
  • User tests assume sometimes less accurate results (e.g. user test: 8.006001 trevas: 8.006000999999998)

I think every language will give this kind of result. To be sure of the result, if a particular precision is needed, user can use round operator. @hadrienk, can you confirm?

@NicoLaval NicoLaval added question Further information is requested Needs more information and removed question Further information is requested labels May 20, 2024
@noahboerger
Copy link
Collaborator

I fully agree on this. This is not an issue on trevas, but more on the testcases we have been using. I think these should be adjusted to use the rounding operation where necessary.

@hadrienk
Copy link
Collaborator

@NicoLaval yes, rounding is the solution. This is a case of the good old 0.1 + 0.2 != 0.3.
https://0.30000000000000004.com/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants