Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve performance of data loading in web app #1757

Open
jacobthill opened this issue Mar 14, 2023 · 0 comments
Open

Improve performance of data loading in web app #1757

jacobthill opened this issue Mar 14, 2023 · 0 comments

Comments

@jacobthill
Copy link
Contributor

jacobthill commented Mar 14, 2023

Currently it takes a very long time to load data in stage (probably prod as well but we can't test this until the data share drive issue is resolved). Loading 20k records takes approximately 24 hours. This makes data aggregation very slow.

To reproduce the issue, transform a set of records locally (see dlme-transform Readme) and send them to the server

scp [~/Path/To/NDJson/File] [email protected]:/opt/app/dlme/datashare

Then navigate to https://dlme-stage.stanford.edu/library/catalog/admin select Add items, enter the name of the transformed file (e.g. aub-output.ndjosn) in the box and click Fetch.

Can we investigate why it takes so long and see if we can speed it up?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant