fix: address multiquery long timeouts causing excessive queue growth … #3284
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
stbrody
reviewed
Oct 10, 2024
| @@ -67,7 +67,7 @@ import { LevelKVFactory } from './store/level-kv-factory.js' | |||
| const METRICS_CALLER_NAME = 'js-ceramic' | |||
| const DEFAULT_CACHE_LIMIT = 500 // number of streams stored in the cache | |||
| const DEFAULT_QPS_LIMIT = 10 // Max number of pubsub query messages that can be published per second without rate limiting | |||
| const DEFAULT_MULTIQUERY_TIMEOUT_MS = 30 * 1000 | |||
| const DEFAULT_MULTIQUERY_TIMEOUT_MS = 3 * 1000 | |||
There was a problem hiding this comment.
I'm not positive, but I believe when this timeout fires it will only fail the multiquery request but not clean up the work happening in the background in the execution queue. I think probably you want to pair this with also turning down the IPFS timeouts in the dispatcher.
There was a problem hiding this comment.
ok i had made a separate PR for those since i only saw the multiquery errors in the logs. we could do #3285 instead. I thought that might require more testing if its hitting more locations.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
During a period of heavy traffic and possible IPFS overload or failures, many errors of the following form occurred:
Approximately 292K in one day.
We also have been seeing dramatic queue size growth and p-queue memory usage during these periods of intense load. This could be related to the large number of streams waiting for the 90 sec timeout, which still eventually fails. It would be better to fail sooner and not increase the load further.
Include relevant motivation, context, brief description and impact of the change(s). List follow-up tasks here.
How Has This Been Tested?
Describe the tests that you ran to verify your changes. Provide instructions for reproduction.
PR checklist
Before submitting this PR, please make sure:
References:
Please list relevant documentation (e.g. tech specs, articles, related work etc.) relevant to this change, and note if the documentation has been updated.