Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add OperationsPreprocessor to queueing #1485

Merged
merged 10 commits into from
Oct 6, 2023
Merged

Add OperationsPreprocessor to queueing #1485

merged 10 commits into from
Oct 6, 2023

Conversation

BartoszPrusak
Copy link
Contributor

Adding OperationsPreprocessor to AsyncOperationProcessor so that:

  • number of actual (efficiently combined) operations can be used to limit the size of a batch,
  • number of series appends can be used to limit the size of a batch.

@BartoszPrusak
OperationsPreprocessor tracks number of operations and appends
6b12193 
This will be used to get better tracking whether an operation batch
should be sent or not yet.
@BartoszPrusak
Resolve Python 3.7 compatibility issue
c1aa184 
`type` is not subscribable as `type[Foo]` until Python 3.9. For older
Python versions `typing.Type[Foo]` needs to be used.
@BartoszPrusak
OperationPreprocessor.process() should not throw RequiresPreviousComp…
5c587e2 
…leted
@BartoszPrusak
Fix List[] vs list[]
254fd6b
@BartoszPrusak
Use OperationsPreprocessor to regulate the size of a batch
89bb819
@BartoszPrusak
Fix Final not available in Python 3.7
f64deea
@BartoszPrusak BartoszPrusak requested a review from Raalsky October 5, 2023 09:25
@BartoszPrusak BartoszPrusak changed the base branch from dev/new-batching to master October 5, 2023 11:43
@codecov
Copy link

codecov bot commented Oct 5, 2023 •
edited
Loading

Codecov Report

Attention: 3 lines in your changes are missing coverage. Please review.

Files Coverage Δ
...eptune/internal/backends/hosted_neptune_backend.py 68.04% <100.00%> (-10.12%) ⬇️
...ptune/internal/backends/operations_preprocessor.py 96.84% <100.00%> (+<0.01%) ⬆️
src/neptune/internal/operation.py 96.41% <88.88%> (-0.19%) ⬇️
.../operation_processors/async_operation_processor.py 86.11% <93.93%> (+1.08%) ⬆️

... and 53 files with indirect coverage changes

📢 Thoughts on this report? Let us know!.

@Raalsky Raalsky merged commit edc28ac into master Oct 6, 2023
4 checks passed
@Raalsky Raalsky deleted the bp/better-batches branch October 6, 2023 09:01
aniezurawski added a commit that referenced this pull request Oct 9, 2023
@aniezurawski
Revert "Add OperationsPreprocessor to queueing (#1485)"
c2a19e0 
This reverts commit edc28ac.
Raalsky pushed a commit that referenced this pull request Oct 9, 2023
@BartoszPrusak @Raalsky
Add OperationsPreprocessor to queueing (#1485)
3213b90
PatrykGala added a commit that referenced this pull request Oct 10, 2023
@PatrykGala
Revert "Add OperationsPreprocessor to queueing (#1485)"
b737e26 
This reverts commit 3213b90.
PatrykGala added a commit that referenced this pull request Oct 10, 2023
@PatrykGala
Revert "Revert "Add OperationsPreprocessor to queueing (#1485)""
b134de0 
This reverts commit b737e26.
Raalsky pushed a commit that referenced this pull request Oct 10, 2023
@BartoszPrusak @Raalsky
Add OperationsPreprocessor to queueing (#1485)
61c97ed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Raalsky Raalsky Raalsky approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@BartoszPrusak @Raalsky