Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Does not find correct file in every case #3

Open
DaniJobe opened this issue Aug 29, 2016 · 0 comments
Open

Does not find correct file in every case #3

DaniJobe opened this issue Aug 29, 2016 · 0 comments

Comments

@DaniJobe
Copy link

DaniJobe commented Aug 29, 2016

If the difference between desired monitor resolution and the file's resolution is too large the I16 that stores the "min error" in "Find Best XML File In Folder" rolls over (sometimes to a negative number) and in some cases results in returning the wrong file - not the one containing the minimum error.

For example, I have two files corresponding to two screen resolutions - one for 2560x1440 and the other for 1596x1127. If I am currently using a monitor that has 2560x1440 I expect it to return the file for 2560x1440.
2016-08-29_13-36-02
The difference between the 1592x1127 file and my 2560x1440 resolution is 964x313 and when it is squared it becomes 11792x32433. Those numbers are added and compared to the minimum error to determine whether it is the new minimum or not. Since it is an I16 and 11792+32433=44225 which is > 32767, it rolls over and returns -21311. Since this is (incorrectly) smaller than the current error, it is storing this value as the minimum and passing out the file index corresponding to this incorrect minimum instead of the one with the actual smaller number.

When I change the number to a 32-bit integer this problem goes away.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant