Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speedup insert #96

Merged
merged 5 commits into from
Aug 29, 2023
Merged

Speedup insert #96

merged 5 commits into from
Aug 29, 2023

Conversation

soad003
Copy link
Member

@soad003 soad003 commented Aug 25, 2023

The PR looks great to me. I have added the following:

  • parameters to control the number of workers (--n-workers; defaults to 1 so parallel is opt-in, 0, and negative numbers can be used to as offset to cpu_count, positive values just specify the nr of workers)
  • added opt-out validation to the tagpack insertion; not really related to the PR but I was startled that this was not already done
  • added retry handling for deadlock situations.
  • added a single thread mode that runs the insert in the main thread as before (makes pdb debugging way easier)
  • added code to share the tagstore connection (one per worker process). Otherwise, every tagpack established a new connection to the database.

@soad003
Copy link
Member Author

soad003 commented Aug 25, 2023

I will test some more but i think this is ready to be merged! Great work @mdragaschnig

Copy link
Member

@mdragaschnig mdragaschnig left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good.

@mdragaschnig mdragaschnig merged commit 8cd22b1 into develop Aug 29, 2023
@mdragaschnig mdragaschnig deleted the speedup_insert branch August 29, 2023 10:55
@soad003 soad003 mentioned this pull request Aug 29, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants