Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add pre-analysis hook to new workflow #634

Closed
sambles opened this issue May 30, 2022 · 1 comment · Fixed by #663
Closed

Add pre-analysis hook to new workflow #634

sambles opened this issue May 30, 2022 · 1 comment · Fixed by #663
Assignees
Milestone

Comments

@sambles
Copy link
Contributor

sambles commented May 30, 2022

The pre-analysis hook needs integration with the new platform

The distributed worker is calling the oasislmf manager function OasisManager().generate_files(**params) to create the oasis run files. This doesn't include the pre-analysis hook step.

The older worker used something equivalent to OasisManager().generate_oasis_files(**params) from oasislmf/computation/run/generate_files.py which conditionally calls ExposurePreAnalysis.

Swapping one for the other might be problematic, since at this point in the execution the distributed worker has already generated lookup results with the unmodified location file.

An optional step or task is needed before the sub-task(s) prepare_keys_file_chunk generates lookup results

@sambles
Copy link
Contributor Author

sambles commented Jun 14, 2022

Two options for adding:

  1. add new optional task to run pre-analysis hook before keys lookup. Then push the modified location file back in the server's storage and continue with lookup stage.

  2. Run each pre-analysis as a partial step before running the lookup chunk. Combine all partial location data back into a single file and push that into sever storage once the all lookup's have been processed.

modified exposure storage
a) Copy an existing portfolio and store the modified files under a new portfolio id, have both portfolio links under an analysis object.

b) Add new endpoints to all portfolio's for modified file storage.

@sambles sambles linked a pull request Jul 1, 2022 that will close this issue
@sambles sambles closed this as completed Jul 5, 2022
@awsbuild awsbuild added this to the 2.1.1 milestone Apr 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants