Features : Optional_Files, jobTimeout and LogStream to metadata #55
+719
−174
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Optional Files
As raised by Peter Thomas on slack :
Added support for optional files in the AWS handling of (de)localization and caching:
Tested for input and output of individual tasks and workflows:
Note: Mixed types are not supported and cast to mandatory files:
jobTimeout
It is now possible to specify a maximal job runtime (walltime) for jobs on the AWS backend. If the time is exceeded, the job is teminated. Use this to kill hanging jobs (seen in R multicore processing in our case). Added to README for documentation
LogStream
The CloudWatch LogGroupName, LogStreamName and the AWS region the job was executed in, are now added to the task call metadata. Query cromwell for metadata to retrieve it. These logstreams also contain info on (de)-localization, in contstrast to stdout/stderr from the metadata.
extra : minor optimization on hashing through EFS/MD5 files : if considered invalid, return a random string instead of the same message each time. This forces a cache-break.
For testing : see release 87.1-AWS in my own fork : https://github.com/geertvandeweyer/cromwell/releases/tag/87.1-AWS