Optimize the JSON parsing in NpmPackageIndexBuilder.seeFile #1898
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
qligier
force-pushed
the
ql_optimize_npib
branch
from
544c208 to
c63277c
Compare
|
The R4B test failures look related to this |
qligier
force-pushed
the
ql_optimize_npib
branch
from
c63277c to
ec8d461
Compare
qligier
force-pushed
the
ql_optimize_npib
branch
from
c3603d3 to
053d201
Compare
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #1898 +/- ##
=========================================
Coverage 12.91% 12.91%
- Complexity 34012 34021 +9
=========================================
Files 2252 2252
Lines 686541 686547 +6
Branches 202569 202576 +7
=========================================
+ Hits 88643 88650 +7
+ Misses 566487 566483 -4
- Partials 31411 31414 +3 ☔ View full report in Codecov by Sentry. |
|
Yep, I fixed it. |
dotasek
approved these changes
Feb 6, 2025
hl7-devops
added a commit
that referenced
this pull request
Feb 9, 2025
## Announcement This release marks the beginning of a process of refactoring the HAPI core code. We are trimming and refactoring the core model code to reduce dependencies. As part of this, we will be culling all the old unmaintained code in the older versions. In addition, we will be moving all the terminology, rendering, view definition, and validation related code to a new partner package for R4, R4B, and R5. This change is planned for July 2025. This release starts the process of marking the code with annotations to indicate it's proposed fate: - @deprecated classes will be deleted in July 2025 unless users raise issues with that - @MarkedToMoveToAdjunctPackage is code that will move the other package - Code with no annotations will not move or be deleted ## Validator Changes * Add HL7 CodeSystem display and definition checks * Add Matchetype validator * Add "http://hl7.org/fhir/tools/StructureDefinition/snapshot-base-version" to snapshot generation * Optimize the JSON parsing in NpmPackageIndexBuilder.seeFile (#1898) (faster loading) * Fix stack crash when structure definitions are circular * Fix error reporting duplicate contained IDs when contained resources are sliced by a profile * Allow cardinality changes in obligation profiles (but not recommended) * Fix bug with wrongly processing -ips#(v) parameter * Add underscore to regex to be able to use underscore in Bundle URLs ## Other code changes * Refactor FileUtilities and other Utilities classes * fix element order in Element.forceElement() * fix NPE in patient renderer * Resource Factory updates for loading generated resources in IG publisher * Fix intermittent thread issue in Date rendering ***NO_CI***
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
While working on some performance issues in Matchbox, I noticed that
NpmPackageIndexBuilder.seeFilewas using a lot of resources (time and memory allocation). The reason is that it fully parses many JSON files, allocating memory for objects representing the whole file, just to read a few properties and discard the object immediately.This is a use case where streaming parsers shine: they only parse and allocate memory for the parts we're interested in.
This PR uses the streaming parser provided in Jackson.
In Matchbox startup, we measured a reduction of ~6 seconds (12%) and ~7 Go of allocation (23%), when loading
hl7.fhir.r4.core,hl7.terminology.r4andhl7.fhir.uv.extensions.r4.