Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Backend NML Parser: Performance Improvements #4872

Merged
merged 4 commits into from
Oct 19, 2020
Merged

Backend NML Parser: Performance Improvements #4872

merged 4 commits into from
Oct 19, 2020

Conversation

fm3
Copy link
Member

@fm3 fm3 commented Oct 14, 2020

I had wrongly assumed the bulk of the upload nml complexity to be in validateTrees.

Analysis with a 350 MB real-world NML yielded:

45 s  load xml from input stream
961 s  parse trees
3 s  splitMultiComponentTrees
6 s  validateTrees
0.7 s  send to tracingstore & save

This PR tackles two properties of parseTrees:
(1) the accessing of XMLNode attributes via shortcut method \ is not very efficient, new helper method getSingleAttribute (speedup factor ×2–4)
(2) replace inefficient handling of comments and branchpoints by hashmap lookup (speeup factor ×10–15 for this particular NML, which notably has 80k node comments)

so the parse trees line after this change reads with speedup factor ×36

27 s  parse trees

with the others unchanged.

Other changes:

  • increase request timeouts from 1000s (16.6min) to 10000s (166min)
  • use the same request timeouts in dev mode (has to be defined in built.sbt)
  • when killing the proxy.js, send sigkill rather than sigterm to child processes, avoiding port-in-use problems

TODO

  • use getSingleAttribute in remaining places (won’t bring much more speedup, just for consistency)
  • cleanup (remove time-logging, re-enable startup scripts etc)

Steps to test:

  • upload very large NML via dashboard
  • should be faster, should yield the same SkeletonTracing

@fm3 fm3 self-assigned this Oct 14, 2020
@fm3 fm3 changed the title [WIP] backend nml parser performance improvements Backend NML Parser: Performance Improvements Oct 15, 2020
@fm3 fm3 requested a review from youri-k October 15, 2020 11:28
Copy link
Contributor

@youri-k youri-k left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM :shipit:

@fm3 fm3 merged commit 1d8f0c3 into master Oct 19, 2020
@fm3 fm3 deleted the nml-backend-perf branch October 19, 2020 11:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants