Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rework arange handling of uint and bigint arguments #2603

Closed
stress-tess opened this issue Jul 21, 2023 · 1 comment · Fixed by #2604
Closed

rework arange handling of uint and bigint arguments #2603

stress-tess opened this issue Jul 21, 2023 · 1 comment · Fixed by #2604
Assignees
Labels
Technical Debt Rework code for more flexibility or readability

Comments

@stress-tess
Copy link
Member

stress-tess commented Jul 21, 2023

We need to rework arange, right now the chapel code assumes if the arguments aren't bigint that they are int. This leads to server crash when arguments are uints in the 2**63..2**64 range

for example:

# fits in an int, so it works
>>> ak.arange(15)
array([0 1 2 3 4 5 6 7 8 9 10 11 12 13 14])

# in bigint range, only works if start and stop are specified
>>> ak.arange(2**64, 2**64 + 15)
array(["18446744073709551616" "18446744073709551617" "18446744073709551618" "18446744073709551619" "18446744073709551620" "18446744073709551621" "18446744073709551622" "18446744073709551623" "18446744073709551624" "18446744073709551625" "18446744073709551626" "18446744073709551627" "18446744073709551628" "18446744073709551629" "18446744073709551630"])

# in bigint range but only stop specified, doesn't error but gives incorrect results
>>> ak.arange(2**64 + 15)
array(["0" "1" "2" "3" "4" "5" "6" "7" "8" "9" "10" "11" "12" "13" "14"])

# upper end of uint, start and stop specified is treated as int and gives incorrect results
>>> ak.arange(2**63, 2**63 + 15)
array([-9223372036854775808 -9223372036854775807 -9223372036854775806 -9223372036854775805 -9223372036854775804 -9223372036854775803 -9223372036854775802 -9223372036854775801 -9223372036854775800 -9223372036854775799 -9223372036854775798 -9223372036854775797 -9223372036854775796 -9223372036854775795 -9223372036854775794])

# in upper end of uint, only stop specified causes server crash
>>> ak.arange(2**63 + 15)

the server crashes with this error message

/Users/pierce/proj/arkouda//src/SymArrayDmap.chpl:58: error: Attempting to allocate > max(size_t) bytes of memory

I think we can get around this by using resolve_scalar_dtype to determine the type of the arguments. Then we can either pass this to the chpl side and do a select or just get this info from message args

It's worth noting this isn't terrible to workaround (we can just add the 2**63 after the fact), but this should def be handled correctly

>>> ak.arange(15, dtype=ak.uint64) + 2**63
array([9223372036854775808 9223372036854775809 9223372036854775810 9223372036854775811 9223372036854775812 9223372036854775813 9223372036854775814 9223372036854775815 9223372036854775816 9223372036854775817 9223372036854775818 9223372036854775819 9223372036854775820 9223372036854775821 9223372036854775822])

>>> ak.arange(15) + 2**64
array(["18446744073709551616" "18446744073709551617" "18446744073709551618" "18446744073709551619" "18446744073709551620" "18446744073709551621" "18446744073709551622" "18446744073709551623" "18446744073709551624" "18446744073709551625" "18446744073709551626" "18446744073709551627" "18446744073709551628" "18446744073709551629" "18446744073709551630"])
@stress-tess stress-tess added bug Something isn't working Technical Debt Rework code for more flexibility or readability labels Jul 21, 2023
@stress-tess stress-tess self-assigned this Jul 21, 2023
@stress-tess stress-tess changed the title bug in arange handling of uint arguments >2**63 bug in arange handling of uint and bigint arguments Jul 21, 2023
@stress-tess
Copy link
Member Author

stress-tess commented Jul 21, 2023

Oh i think i'm being dumb. We def needed to rework this but 2**63 + 15 is just a huge number, so my computer doesn't have nearly enough memory lol

@stress-tess stress-tess removed the bug Something isn't working label Jul 21, 2023
@stress-tess stress-tess changed the title bug in arange handling of uint and bigint arguments rework arange handling of uint and bigint arguments Jul 21, 2023
stress-tess pushed a commit to stress-tess/arkouda that referenced this issue Jul 21, 2023
This PR (closes Bears-R-Us#2603) rework how arange handles `uint` and `bigint` arguments. This update provides the correct return for something in the `> 2**63` range such as `ak.arange(2**63, 2**63+15)`
stress-tess pushed a commit to stress-tess/arkouda that referenced this issue Jul 21, 2023
This PR (closes Bears-R-Us#2603) rework how arange handles `uint` and `bigint` arguments. This update provides the correct return for something in the `> 2**63` range such as `ak.arange(2**63, 2**63+15)`
stress-tess pushed a commit to stress-tess/arkouda that referenced this issue Jul 21, 2023
This PR (closes Bears-R-Us#2603) rework how arange handles `uint` and `bigint` arguments. This update provides the correct return for something in the `> 2**63` range such as `ak.arange(2**63, 2**63+15)`
github-merge-queue bot pushed a commit that referenced this issue Jul 24, 2023
This PR (closes #2603) rework how arange handles `uint` and `bigint` arguments. This update provides the correct return for something in the `> 2**63` range such as `ak.arange(2**63, 2**63+15)`

Co-authored-by: Pierce Hayes <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Technical Debt Rework code for more flexibility or readability
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant