Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Speed up address-to-cluster mapping #4

Closed
mdragaschnig opened this issue Dec 1, 2021 · 5 comments
Closed

Speed up address-to-cluster mapping #4

mdragaschnig opened this issue Dec 1, 2021 · 5 comments
Assignees
@soad003 @mdragaschnig

Comments

@mdragaschnig
Copy link
Member

No description provided.

@defconst defconst changed the title Speed up validation Speed up cluster mapping Oct 9, 2022
@behas
Copy link
Member

behas commented Oct 27, 2022

@soad003 plz review the current implementation and share your ideas on how to speed up the mapping.

@behas behas changed the title Speed up cluster mapping Speed up address-to-cluster mapping Oct 27, 2022
@soad003
Copy link
Member

soad003 commented Oct 27, 2022

will do

@mdragaschnig
Copy link
Member Author

Parallelise

https://github.com/graphsense/graphsense-tagpack-tool/blob/develop/bin/tagpack-tool#L399

@soad003
Copy link
Member

soad003 commented Nov 15, 2022 •
edited
Loading

Code now executes address batches (default 5000) in parallel.

graphsense-tagpack-tool/bin/tagpack-tool

Line 401 in a5b5b3d

def insert_cluster_mapping(args, batch_size=5_000):

@mdragaschnig please review.

@mdragaschnig
Copy link
Member Author

During my tests, speedup was more than fourfold, nice! (6 workers vs. 1 worker)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@soad003 soad003

@mdragaschnig mdragaschnig

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@behas @soad003 @mdragaschnig