Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

execute update query in batch to avoid slow bulk action query #3280

Merged
merged 6 commits into from
Dec 29, 2023

Conversation

sywhb
Copy link
Contributor

@sywhb sywhb commented Dec 26, 2023

No description provided.

Copy link

vercel bot commented Dec 26, 2023

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
omnivore-demo ✅ Ready (Inspect) Visit Preview 💬 Add feedback Dec 29, 2023 8:45am
omnivore-prod ✅ Ready (Inspect) Visit Preview 💬 Add feedback Dec 29, 2023 8:45am

@jacksonh
Copy link
Contributor

0_0

Was going to ask about this as some of the clean up features will rely heavily on bulk.

@sywhb
Copy link
Contributor Author

sywhb commented Dec 28, 2023

@jacksonh i think this still might not be the best way to achieve incremental update but that's so far the best way i can do:

in the db transaction:

  • Get the total count of rows to be updated
  • Calculate the number of batches
  • Open a cursor
  • Loop through batches
  • Fetch the next batch of IDs
  • Perform incremental update for the current ID
  • Close the cursor

let me know if you got some better idea or alternatives 👍

@sywhb
Copy link
Contributor Author

sywhb commented Dec 28, 2023

similar idea could be applied to prune user accounts -- incremental delete of library items and then delete user account

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants