Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove N+1 in sync #326

Open
lmatayoshi opened this issue Nov 25, 2020 · 0 comments
Open

Remove N+1 in sync #326

lmatayoshi opened this issue Nov 25, 2020 · 0 comments
Labels
collector Issues for maap-collector performance

Comments

@lmatayoshi
Copy link
Contributor

lmatayoshi commented Nov 25, 2020

At the moment, N+1 queries are being triggered when fetching entries from the database. In remoteSync function, the mapping forces an individual query for each fetched record, which results in a lot of processing time. This situation becomes really annoying in environments such as Nigeria, which at the moment accounts for 70.000+ AntibioticConsumptionStats.

network_devtools

We can see in the picture taken from the Network tab in the dev tools that fetching data from the server only takes about 1 second, whereas resolving the queries takes 10 times more time: 10+ seconds.

Solution involves turning that mapping of individual queries into a single one that creates or updates all the entries at once. Double check that synchronization keeps working after the fix, because the process has its quirks and it easily gets broken

@lmatayoshi lmatayoshi added store (admin) Issues for maap-store admin collector Issues for maap-collector performance and removed store (admin) Issues for maap-store admin labels Nov 25, 2020
@lmatayoshi lmatayoshi changed the title Reduce antibiotic consumption sync time Remove N+1 in sync Nov 30, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
collector Issues for maap-collector performance
Projects
None yet
Development

No branches or pull requests

1 participant