Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Break down large PouchDB.bulkDocs calls #2362

Merged
merged 1 commit into from
Dec 10, 2024

Conversation

taratatach
Copy link
Member

@taratatach taratatach commented Nov 18, 2024

Large bulkDocs calls (i.e. with thousands of documents to save at
once) can be the source of crashes.

We're not sure if this comes from PouchDB or the logger calls with
each document but we've figured that creating smaller batches of
documents, logging them all at once and then saving the batch in
PouchDB prevents crashes.

It should also result in less I/O since we're making a lot less logger
calls.
To accommodate for these logging changes, we update our jq path
filter so it looks into batches for documents with the expected path.

Please make sure the following boxes are checked:

  • PR is not too big
  • it improves UX & DX in some way
  • it includes unit tests matching the implementation changes
  • it includes scenarios matching a new behaviour or has been manually tested
  • it includes relevant documentation

@taratatach taratatach self-assigned this Nov 18, 2024
@taratatach taratatach force-pushed the fix/break-down-large-pouchdb-bulk-docs-save branch from b1fecbe to da7e156 Compare December 10, 2024 16:18
@taratatach taratatach changed the title WIP fix: Break down large PouchDB.bulkDocs calls Dec 10, 2024
@taratatach taratatach force-pushed the fix/break-down-large-pouchdb-bulk-docs-save branch from da7e156 to 797af62 Compare December 10, 2024 16:19
@taratatach taratatach marked this pull request as ready for review December 10, 2024 16:19
  Large `bulkDocs` calls (i.e. with thousands of documents to save at
  once) can be the source of crashes.

  We're not sure if this comes from PouchDB or the logger calls with
  each document but we've figured that creating smaller batches of
  documents, logging them all at once and then saving the batch in
  PouchDB prevents crashes.

  It should also result in less I/O since we're making a lot less logger
  calls.
  To accommodate for these logging changes, we update our `jq` `path`
  filter so it looks into batches for documents with the expected path.
@taratatach taratatach force-pushed the fix/break-down-large-pouchdb-bulk-docs-save branch from 797af62 to 3579de3 Compare December 10, 2024 16:37
@taratatach taratatach requested a review from nono December 10, 2024 16:37
@taratatach taratatach merged commit 0e5b89b into master Dec 10, 2024
13 of 16 checks passed
@taratatach taratatach deleted the fix/break-down-large-pouchdb-bulk-docs-save branch December 10, 2024 17:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants