Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix performance degradation in ignite streaming endpoint #516

Merged
merged 7 commits into from
Sep 29, 2023

Conversation

tyrasd
Copy link
Member

@tyrasd tyrasd commented Sep 29, 2023

#495 had unfortunately introduced a bug1 in affinitycall's stream implementation which caused the backend to always use the slow fallback path (i.e. .collect().stream()). This is particularily slow for medium to large bounding boxes, as it will not use the preflight optimizations implemented for stream.

For example, this has the effect that with this version all ohsome-api data extractions are very slow.

Checklist

Footnotes

  1. the concrete problem being that stream.toList() returns an immutable list which cannot be shuffled in the following code line, while the previously used stream.collect(Collectors.toList()) does return a mutable list (even though it does not guarantee the mutability of it, btw).

this bug caused the backend to always use the slow path of `.collect().stream()`, which is particularily slow for medium to large bounding boxes, as it will not use the preflight optimizations implemented for `stream`.
@tyrasd tyrasd added bug Something isn't working as expected performance Performance optimizations, bottlenecks of the current pipeline, etc. labels Sep 29, 2023
@tyrasd tyrasd changed the title Fix performance degregation in ignite streaming endpoint Fix performance degradation in ignite streaming endpoint Sep 29, 2023
@tyrasd tyrasd requested a review from rtroilo September 29, 2023 11:10
Copy link
Member

@rtroilo rtroilo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thank you!

@tyrasd tyrasd merged commit 6e5cab8 into master Sep 29, 2023
1 check passed
@tyrasd tyrasd deleted the fix-streaming-endpoint branch September 29, 2023 12:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working as expected performance Performance optimizations, bottlenecks of the current pipeline, etc.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants