Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sporadic failures of Parquet segarray_write test with gasnet #2268

Closed
bmcdonald3 opened this issue Mar 21, 2023 · 4 comments · Fixed by #2534
Closed

Sporadic failures of Parquet segarray_write test with gasnet #2268

bmcdonald3 opened this issue Mar 21, 2023 · 4 comments · Fixed by #2534
Assignees
Labels
bug Something isn't working In Progress Work on ticket is in progress / ticket is actively being worked

Comments

@bmcdonald3
Copy link
Contributor

Describe the bug
This failure:

        # bool with empty segments
        a = [0, 1, 1]
        c = [1, 0]
        s = ak.SegArray.from_parts(ak.array([0, 0, len(a), len(a), len(a), len(a) + len(c)]), ak.array(a + c,
                                   dtype=ak.bool))
        with tempfile.TemporaryDirectory(dir=ParquetTest.par_test_base_tmp) as tmp_dirname:
            s.to_parquet(f"{tmp_dirname}/int_test")

            rd_data = ak.read_parquet(f"{tmp_dirname}/int_test*")
            for i in range(6):
>               self.assertListEqual(s[i].tolist(), rd_data[i].tolist())
E               AssertionError: Lists differ: [False, True, True] != []
E
E               First list contains 3 additional elements.
E               First extra element 0:
E               False
E
E               - [False, True, True]
E               + []

tests/parquet_test.py:483: AssertionError

popped up on nightly testing. I am able to reproduce this issue on 16 locales, but it appears to only show up about once in every 100 trials.

To Reproduce

  • Edit pytest.ini to run with 4 or more locales (not sure if this is necessary, but I couldn't reproduce with 2 locales, but could with 16; fairly confident anything 4 or over would work based on the type of failure and what we've seen here in the past)
  • Modify test_segarray_write to run a large number of trials (I seem to catch this about 1 in every 100 runs)
  • Run python3 -m pytest tests/parquet_test.py -k "test_segarray_write" to run just this test
@bmcdonald3 bmcdonald3 added the bug Something isn't working label Mar 21, 2023
@Ethan-DeBandi99 Ethan-DeBandi99 self-assigned this Mar 21, 2023
@Ethan-DeBandi99 Ethan-DeBandi99 added the In Progress Work on ticket is in progress / ticket is actively being worked label Mar 21, 2023
@Ethan-DeBandi99
Copy link
Contributor

Had conversation with @bmcdonald3 documenting here for awareness. @pierce314159 and I were unable to replicate the issue. We will continue to monitor this to hopefully determine what is causing it.

@Ethan-DeBandi99 Ethan-DeBandi99 removed the In Progress Work on ticket is in progress / ticket is actively being worked label Mar 28, 2023
@Ethan-DeBandi99
Copy link
Contributor

I believe I located the issue causing this. While working Issue #2493, I discovered that there is an issue when a locale has no values being written. In writeSegArrayComponent in ParquetMsg.chpl, if the localVals variable has a size 0, there is an error.

@Ethan-DeBandi99 Ethan-DeBandi99 added the In Progress Work on ticket is in progress / ticket is actively being worked label Jun 23, 2023
@Ethan-DeBandi99
Copy link
Contributor

I corrected the first layer here, but it appears that there are more issues occurring that are related.

@Ethan-DeBandi99
Copy link
Contributor

Through testing I am able to replicate issue with 3 locales running this test. However, based on the error I am seeing it appears to be an allocation error. I am working to track this down, but it is proving rather tricky to locate.

github-merge-queue bot pushed a commit that referenced this issue Jul 6, 2023
…st with gasnet (#2534)

* Final correction for issue. Error is resolved, code still requires cleanup.

* Final updates to prevent memory allocation errors.

* Clean up

* Removing prints from test. Adding comments for early outs.

* Removing prints
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working In Progress Work on ticket is in progress / ticket is actively being worked
Projects
None yet
2 participants