Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize load of ERC20 addresses #2324

Merged
merged 1 commit into from
Nov 14, 2024
Merged

Optimize load of ERC20 addresses #2324

merged 1 commit into from
Nov 14, 2024

Conversation

Uxio0
Copy link
Member

@Uxio0 Uxio0 commented Nov 11, 2024

Use database iterators to load ERC20 addresses, so only chunks are loaded in RAM, instead of the whole queryset. Using production database I analyzed good values for chunk_size value:

        Chunk size optimization
        -----------------------
        Testing with 3M Safes
        2k - 90 seconds
        100k - 73 seconds
        500k - 60 seconds
        1M - 60 seconds
        3M - 60 seconds
        
        Testing with 15M Safes
        50k - 854 seconds
        500k - 460 seconds
        750k - 415 seconds
        1M - 430 seconds
        2M - 398 seconds
        3M - 407 seconds
        
        500k sounds like a good compromise memory/speed wise

@Uxio0 Uxio0 requested a review from a team as a code owner November 11, 2024 13:26
@Uxio0 Uxio0 marked this pull request as draft November 11, 2024 15:41
@Uxio0 Uxio0 force-pushed the optimize-erc20-addresses branch from 5ac6718 to 7d3b289 Compare November 14, 2024 15:33
- Improve RAM utilization using a database iterator
@Uxio0 Uxio0 force-pushed the optimize-erc20-addresses branch from 7d3b289 to b1f32b0 Compare November 14, 2024 15:42
@Uxio0 Uxio0 marked this pull request as ready for review November 14, 2024 15:42
@Uxio0 Uxio0 merged commit fa72c03 into main Nov 14, 2024
9 checks passed
@Uxio0 Uxio0 deleted the optimize-erc20-addresses branch November 14, 2024 16:17
@github-actions github-actions bot locked and limited conversation to collaborators Nov 14, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Improve memory when loading Safe addresses for ERC20 indexing
2 participants