Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

22.03 Performance Release #1211

Merged
merged 71 commits into from
Mar 23, 2022
Merged

Conversation

ericenns
Copy link
Member

@ericenns ericenns commented Feb 25, 2022

Description of changes

This PR started off as just fixing a scalability issue in ProjectEventHandler and has since transformed into a performance release that enables Users to have massive projects (i.e. >=100,000 samples).

The original Scalability issue that I identified was that as the number of Samples increased in a Project, certain operations would increase in time. An example is as follows:

  1. Create a fresh Project
  2. Create 5000 samples in that Project consecutively
    • Note: I was using API methods in irida-uploader to do this (specifically send_sample);

Here are the results of the test:

Experiment Name IRIDA Version IRIDA Uploader Version Cumulative Time
time.exp1 22.01.X 0.7.1 5 hours and 30 minutes
time.exp2 22.03 0.7.1 20 minutes
time.exp3 22.03 0.8 beta 5 minutes

timing

Summary of Changes:

  • [All]: Fixed scalability bug in updating Project modifiedDate during handling of ProjectEvent that was increasing in operation time as number of Samples increased within a project.
  • [UI]: Fixed load times on Project Samples page.
  • [UI]: Fixed text overflow issue in Command-line Linker modal.
  • [UI]: Fixed bug with Command-line Linker modal making multiple requests to the server to get the command string.
  • [UI]: Fixed issue with CSV/XLSX export on linelist page from not completing on massive projects >100,000 samples.
  • [UI]: Adding in batch loading to linelist page (provides feedback to user, important on large projects >5000 samples)
  • [UI]: Added a progress indicator when loading samples into the linelist table.
  • [UI]: Added a warning notification when selecting all samples on the project samples page.
  • [UI]: Added in missing Illumina, PacBio, and Ion Torrent instruments for NCBI submission.

Related issue

Link to the GitHub issue this pull request addresses using the #issuenum format. If it completes an issue, use Fixes #issuenum to automatically close the issue.

Checklist

Things for the developer to confirm they've done before the PR should be accepted:

  • CHANGELOG.md (and UPGRADING.md if necessary) updated with information for new change.
  • Tests added (or description of how to test) for any new features.
    * [ ] User documentation updated for UI or technical changes.

@ericenns ericenns force-pushed the fix-project-event-scalability-issues branch from 907f6e9 to fe73f5a Compare February 25, 2022 22:09
@ericenns ericenns changed the base branch from development to master February 25, 2022 22:10
@ericenns ericenns force-pushed the fix-project-event-scalability-issues branch from 931db5f to 87cd7e0 Compare March 1, 2022 15:08
ericenns added 2 commits March 1, 2022 09:49
…d calls, and update Sample modifiedDate through custom repository method for speed.
@ericenns ericenns requested a review from deepsidhu85 March 1, 2022 17:13
@ericenns ericenns force-pushed the fix-project-event-scalability-issues branch from 73fc4c2 to 943f527 Compare March 7, 2022 21:26
@ericenns ericenns force-pushed the fix-project-event-scalability-issues branch from 16ec4fe to 5259462 Compare March 22, 2022 14:22
@ericenns ericenns marked this pull request as ready for review March 22, 2022 16:51
@ericenns ericenns changed the title 22.03 Performance Release [22.03] Performance Release Mar 22, 2022
@ericenns ericenns changed the title [22.03] Performance Release 22.03 Performance Release Mar 22, 2022
Copy link
Contributor

@deepsidhu85 deepsidhu85 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is looking reaallyy good! Great work @ericenns @joshsadam. Just a few small things below

Copy link
Contributor

@deepsidhu85 deepsidhu85 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks really good. Thanks guys! That's all from me

@deepsidhu85 deepsidhu85 merged commit e4cad6a into master Mar 23, 2022
@ericenns ericenns deleted the fix-project-event-scalability-issues branch May 27, 2022 12:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants