[kv store] retry unprocessed items #14453
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
3 Ignored Deployments
|
|
Deployment failed with the following error: |
|
@phoenix-o is attempting to deploy a commit to the Mysten Labs Team on Vercel. A member of the Team first needs to authorize it. |
| if let Some(response) = response.unprocessed_items { | ||
| if let Some(unprocessed) = response.into_iter().next() { | ||
| if !unprocessed.1.is_empty() { | ||
| queue.push_back(unprocessed.1); |
There was a problem hiding this comment.
not sure if we want to add backoff here
There was a problem hiding this comment.
Yeah backoff would be great, I wonder if there are settings in dynamodb client which let us configure retries and backoff
There was a problem hiding this comment.
FWIU there's not, it's indeed a little bit confusing to not at least return the error
the reason I'm slightly hesitant about adding backoff is because of the concurrency model in the KV backfiller
we spawn N concurrent actors to process checkpoints, but store one single progress int value in "meta store" for the job
so it's updated only once there's continuation of progress from multiple actors
thus one slow actor can have impact on that metric. I guess we can try it anyway and see if there's any issue with this and it needs to be adjusted
| if let Some(response) = response.unprocessed_items { | ||
| if let Some(unprocessed) = response.into_iter().next() { | ||
| if !unprocessed.1.is_empty() { | ||
| queue.push_back(unprocessed.1); |
There was a problem hiding this comment.
Yeah backoff would be great, I wonder if there are settings in dynamodb client which let us configure retries and backoff
phoenix-o
force-pushed
the
kv_store_retry
branch
from
1c1a9de to
aa76065
Compare
No description provided.