Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[c++] Modify ManagedQuery to perform async queries #1953

Merged
merged 3 commits into from
Dec 6, 2023

Conversation

nguyenv
Copy link
Member

@nguyenv nguyenv commented Dec 5, 2023

Issue and/or context:

Across both APIs we have observed the SOMAArray intermittenly segfault when accessing the array. This occurs due to a race condition where we have multiple read queries and one query closes the array before the other queries have completed. When a query that has not completed tries to access a closed array, it segfaults. The solution is to modify the C++ ManagedQuery so that it performs async queries and waits for all query threads to complete before closing the SOMAArray.

Changes:

johnkerl added a commit that referenced this pull request Dec 5, 2023
Copy link
Contributor

@eddelbuettel eddelbuettel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM but I am not very deep into std::future<> so merely commenting for now.

I am also a little fuzzy about when we actually launch multiple queries at once -- I had the (wrong, clearly) mental model that we launch one, and iterate if its status is incomplete.

Copy link
Member

@johnkerl johnkerl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs to be branched off main and then we can backport to release-1.6

At present it's branched off @mojaveazure 's feature-dev work which is unrelated to, and should not block, the 1.6.0 release we're trying to get out today

@johnkerl
Copy link
Member

johnkerl commented Dec 5, 2023

Meanwhile @nguyenv I really appreciate the debug, and am verifying it on #1943 -- which I will rebase later once this is merged to release-1.6 -- so #1943 will go back to being only apis/r/DESCRIPTION and apis/r/NEWS.md

@eddelbuettel
Copy link
Contributor

Good catch @johnkerl -- I only saw 'one commit' and nodded and was asleep as to the branch it came from.

@nguyenv nguyenv marked this pull request as draft December 5, 2023 22:15
@nguyenv nguyenv force-pushed the viviannguyen/ingest-seurat-with-async-queries branch from f869440 to 2443fc7 Compare December 5, 2023 22:16
@nguyenv nguyenv changed the base branch from paulhoffman/sc-36437/ingest-seurat-s-commands-slot to main December 5, 2023 22:16
@codecov-commenter
Copy link

codecov-commenter commented Dec 5, 2023

Codecov Report

Merging #1953 (0912b26) into main (dbd1089) will decrease coverage by 14.54%.
The diff coverage is n/a.

❗ Current head 0912b26 differs from pull request most recent head 3ab343c. Consider uploading reports for the commit 3ab343c to get more accurate results

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@             Coverage Diff             @@
##             main    #1953       +/-   ##
===========================================
- Coverage   62.81%   48.28%   -14.54%     
===========================================
  Files         106       72       -34     
  Lines       10037     6464     -3573     
===========================================
- Hits         6305     3121     -3184     
+ Misses       3732     3343      -389     
Flag Coverage Δ
python ?
r 48.28% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
python_api ∅ <ø> (∅)
libtiledbsoma ∅ <ø> (∅)

@johnkerl johnkerl mentioned this pull request Dec 5, 2023
@johnkerl johnkerl self-requested a review December 5, 2023 22:38
Copy link
Member

@gspowley gspowley left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! One minor suggestion.

libtiledbsoma/src/soma/managed_query.cc Outdated Show resolved Hide resolved
@nguyenv nguyenv force-pushed the viviannguyen/ingest-seurat-with-async-queries branch from bca4b2b to 3ab343c Compare December 5, 2023 23:35
@nguyenv nguyenv marked this pull request as ready for review December 5, 2023 23:53
@johnkerl johnkerl merged commit 629fa68 into main Dec 6, 2023
12 checks passed
@johnkerl johnkerl deleted the viviannguyen/ingest-seurat-with-async-queries branch December 6, 2023 00:25
github-actions bot pushed a commit that referenced this pull request Dec 6, 2023
* Addition of async queries

* Modify Python CI workflow

* Check validity of `query_future_`
johnkerl pushed a commit that referenced this pull request Dec 6, 2023
* Addition of async queries

* Modify Python CI workflow

* Check validity of `query_future_`

Co-authored-by: nguyenv <[email protected]>
johnkerl added a commit that referenced this pull request Dec 6, 2023
* [r] tiledbsoma-r 1.6.0

* Fix versions in news headers

* try @nguyenv's #1953

* rerun CI

* debug try 3

---------

Co-authored-by: Aaron Wolen <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants