Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flexporter #1219

Merged
merged 36 commits into from
Nov 22, 2023
Merged

Flexporter #1219

merged 36 commits into from
Nov 22, 2023

Conversation

dehall
Copy link
Contributor

@dehall dehall commented Dec 12, 2022

This PR introduces the FHIR Flexible Exporter, aka "Flexporter", seeking public comment. Please let us know if this feature is useful to you, or what we can do to make it more useful.

We get a lot of requests from people who want to use Synthea, but they need some slight changes to the data. For example, they want Appointment resources for every Encounter or another resource type, or they want to filter the data in a certain way, or they want certain fields set on certain resources, or they may want to ensure the output conforms to the IG they are building or working with. The flexporter is meant to allow to allow users to make these kinds of changes to Synthea output without making major changes to the underlying Synthea engine and/or all the modules.

The basis of the Flexporter is a YAML mapping file which contains a series of actions to apply to FHIR bundles as they are exported from Synthea. Full details on all the different actions supported by the flexporter, and how to use them, are on the wiki: https://github.com/synthetichealth/synthea/wiki/Flexporter

From a technical perspective, the Flexporter relies heavily on FHIRPath, in particular HAPI's implementation of the FHIRPath engine. A community contribution to HAPI introduced the FHIRPathResourceGenerator, which I've adopted and modified for the needs of the flexporter. Basically this class takes a mapping of { FHIRPath: value, ... } to populate resources. The most interesting features of the flexporter, including creating new resources and setting values on existing resources, depend on this feature.
There is also a JavaScript engine allowing for arbitrary scripting (though external libraries are not supported)

build.gradle Outdated Show resolved Hide resolved
@codecov
Copy link

codecov bot commented Dec 12, 2022

Codecov Report

Attention: 372 lines in your changes are missing coverage. Please review.

Comparison is base (9f7c6cc) 77% compared to head (de38f5f) 77%.
Report is 29 commits behind head on master.

Files Patch % Lines
src/main/java/RunFlexporter.java 0% 89 Missing ⚠️
...a/org/mitre/synthea/export/flexporter/Actions.java 78% 37 Missing and 34 partials ⚠️
.../mitre/synthea/export/flexporter/FieldWrapper.java 64% 42 Missing and 25 partials ⚠️
.../flexporter/CustomFHIRPathResourceGeneratorR4.java 81% 30 Missing and 9 partials ⚠️
...tre/synthea/export/flexporter/ValueTransforms.java 26% 25 Missing and 3 partials ⚠️
...org/mitre/synthea/helpers/RandomCodeGenerator.java 48% 22 Missing and 6 partials ⚠️
src/main/java/App.java 0% 16 Missing ⚠️
...mitre/synthea/export/flexporter/FhirPathUtils.java 70% 5 Missing and 10 partials ⚠️
...c/main/java/org/mitre/synthea/export/Exporter.java 53% 11 Missing and 2 partials ⚠️
...export/flexporter/FlexporterJavascriptContext.java 78% 4 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             master   #1219     +/-   ##
==========================================
- Coverage        77%     77%     -1%     
- Complexity     3728    3951    +223     
==========================================
  Files           170     178      +8     
  Lines         23981   25360   +1379     
  Branches       3325    3652    +327     
==========================================
+ Hits          18654   19584    +930     
- Misses         4314    4643    +329     
- Partials       1013    1133    +120     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

build.gradle Show resolved Hide resolved
@sonatype-lift
Copy link
Contributor

sonatype-lift bot commented Mar 14, 2023

🛠 Lift Auto-fix

Some of the Lift findings in this PR can be automatically fixed. You can download and apply these changes in your local project directory of your branch to review the suggestions before committing.1

# Download the patch
curl https://lift.sonatype.com/api/patch/github.com/synthetichealth/synthea/1219.diff -o lift-autofixes.diff

# Apply the patch with git
git apply lift-autofixes.diff

# Review the changes
git diff

Want it all in a single command? Open a terminal in your project's directory and copy and paste the following command:

curl https://lift.sonatype.com/api/patch/github.com/synthetichealth/synthea/1219.diff | git apply

Once you're satisfied, commit and push your changes in your project.

Footnotes

  1. You can preview the patch by opening the patch URL in the browser.

build.gradle Show resolved Hide resolved
build.gradle Outdated Show resolved Hide resolved
build.gradle Outdated Show resolved Hide resolved
build.gradle Show resolved Hide resolved
@dehall dehall changed the title WIP: Flexporter Flexporter Aug 11, 2023
@dehall dehall marked this pull request as ready for review August 11, 2023 19:15
Copy link
Collaborator

@hadleynet hadleynet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I built, tested and ran it locally with a mapping file I created by following the wiki instructions.

Code is well structured and there are sufficient tests.

Only a couple of things jumped out at me:

  1. Several Javadoc comments with embedded <> that will be treated as HTML, need to wrap these with {@code ...}
  2. The data structures used in RandomCodeGenerator seem unnecessarily generic. E.g. codeListCache is a Map<String, List<Object>> but the individual list items are typically just cast to Map<String, String>. Consider making this explicit in the declaration of codeListCache as Map<String, List<Map<String, String>>> to avoid all the casts or introduce a new class that inherits from HashMap or something to make the meaning clearer and enforce map keys. E.g. you could use Map<String, List<CodeMap>> whether CodeMap extends or contains a HashMap<String, String> but offers explicit setters and getter for code, code system etc.

@dehall
Copy link
Contributor Author

dehall commented Nov 14, 2023

Thanks @hadleynet , I've made a few additional changes. RandomCodeGenerator now uses Codes internally instead of Map<String,String> and when it loads a ValueSet it uses the HAPI model object instead of working through the JSON via Map<String,Object>. It's still kind of messy but I think it's an improvement

Copy link
Member

@jawalonoski jawalonoski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Below was the test case I used... fields that are "choice" type can be a gotcha when writing FHIR Path (e.g., Procedure.performed[x])... and all the warning messages on stdout are annoying, but otherwise this is good to go.

---
name: Add category to procedures that don't have one

applicability: true

actions:
  - name: Category
    set_values:
      - applicability: Procedure.where(category.empty())
        fields:
          - location: Procedure.category.coding
            value:
              system: http://mitre.org
              code: "XYZZY"
              display: "A flexi category"
  - name: Patient Profile
    profiles:
      - profile: http://fake.org/fhir/Patient/MALE
        applicability: Patient.where(gender="male")
      - profile: http://fake.org/fhir/Patient/FEMALE
        applicability: Patient.where(gender="female")
  - name: testCreateResources_createBasedOn
    create_resource: 
      - resourceType: ServiceRequest
        based_on:
          resource: Procedure
        fields:
          - location: ServiceRequest.intent
            value: plan
          # - location: ServiceRequest.encounter.reference
          #   value: $getField([Procedure.encounter.reference])
          - location: ServiceRequest.authoredOn
            value: $getField([Procedure.performed.start])
          - location: ServiceRequest.subject.reference
            value: $findRef([Patient])
        writeback:
          - location: Procedure.basedOn.reference
            value: $setRef([ServiceRequest])
  - name: testKeepResources
    keep_resources:
      - Patient
      - ServiceRequest

@jawalonoski jawalonoski merged commit 6795013 into master Nov 22, 2023
5 of 6 checks passed
@jawalonoski jawalonoski deleted the flexporter branch November 22, 2023 15:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants