Develop AWS #35
Merged
Develop AWS #35
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Update cromwell version from 83 to 84 * BW-1255 Implement POST /runs endpoint (broadinstitute#6779) * Adding route * Fixing HTTP method error * All formFields made optional * A compliling state * Saving * Saving * All three endpoints functioning as expected; updated RESTAPI.md * Updated response for submission from 200 to 201 to pass tests * Test submission response * Moved updated submission response to askSubmit * test * updating RESTAPI.md * saving * Adding utility file for submitRequest * cleanup * Update awssdkv from 2.17.152 to 2.17.194 (broadinstitute#6814) * BW-1305 Swagger Update (broadinstitute#6818) * Properly documenting metadataArchiveStatus in WorkflowQueryResult model * Update docs * BT-710 Add configs for BlobPathBuilderFactory (broadinstitute#6817) BT-710 Add configs for BlobPathBuilderFactory * BW-1305 Make "name" optional in workflow query response (broadinstitute#6821) * BT-724 Fix BlobPathBuilder failing on retrieving existing filesystem (broadinstitute#6816) Modify blobPathBuilder to fallback to creating a filesystem if one is not found * Logging updates: (broadinstitute#6813) * [BT-698] first pass on BlobTokenGenerator with E2E test (broadinstitute#6824) * first pass on BlobTokenGenerator with E2E test * update BlobPathBuilder constructor args in test * account -> container level client * [BT-687] specify correct types (broadinstitute#6829) * specify correct types * fix test with new type * remove type declarations in function call * remove unnecessary sas-token config * BW-1206 - Combine all Wes Endpoints & add Tests (broadinstitute#6833) * Add tests, getting frid of WesRunRoutes.scala * wesWorkflowId fix, ec implicits errors gone * Refactoring path for GET /runs * Indentation fix * Commit to rollback * Revert "Indentation fix" This reverts commit 63fc484. * PR trigger * Optimize imports * Missed import * BW-1354 - Porting CBAS preliminary step (broadinstitute#6837) * Getting rid of shared utility file; Adding/Updating WES version of submit. * Edit spec file * Adding Wes-like error * BW-1378 Addl CromIAM user enablement checks (broadinstitute#6826) * Update cromwell version from 84 to 85 * BW-1393 Release doc updates (broadinstitute#6839) * BT-732 Checksum validation for blobs read by engine (broadinstitute#6838) * Draft support for optional FileHash * Draft getMd5 for BlobPath * Resolve non-parallel IO to fix tests * Checksum validation for BlobPath * Nicer error message * Test for missing Blob hash * Break attr acquisition into separate method * Cleanup, comments * In-progress tests of blob hash command * Remove test * Remove unused import * BT-711 Refresh SAS token for filesystem on expiry (broadinstitute#6831) * BT-711 Refresh SAS token for filesystem on expiry * Rough cut of token refresh using exceptions * Ignore tests, and minor cleanup * Remove stray line * Draft of manager class for handling expiring file systems * Style fixes * Refactor of blobfilesystemManager and tests covering its functionality * Refined tests to validate close filesystem as separate unit * Ignore connected tests * Clean up of some things * Refactor BlobFileSystemManager to separate file, and some other cleanup * Some additional scala-ifying * Small cleanup * Correcting imports * trigger tests * trigger tests * Batch 1 of scala steward updates (broadinstitute#6903) * Batch 1 of scala steward updates * Rollback snakeYAML * Attempt 3, with only the passing dependancies * Revert google API and Big Query udpates * Winding back other google deps * rollback remaining google updates * trigger tests * trigger tests * [BW-1398] Migrate PKs to BIGINT (broadinstitute#6907) * BT-745 Batch 2 of scala steward updates (broadinstitute#6906) * Update SBT to 2.0.0 * Fix sbt-git import * Update mouse to 1.0.11 * Update rhino 1.7.14 * SUP-692 Retry with more memory after RC 137 (broadinstitute#6912) * Reorder execution result checks so 137 can retry with more memory * Test for memory retry after 137 RC * Fix test expectations * Make memory retry checks consistent * Revert changes to existing test * Rename retryWithMoreMemory to outOfMemoryDetected * Scala steward updates batch 3 (broadinstitute#6913) * Scala steward updates batch 3 * WX-745 Batch 4 scala steward updates (broadinstitute#6916) * WX-746 Localize all DRS inputs in a single Action (broadinstitute#6914) Co-authored-by: Janet Gainer-Dewar <[email protected]> * WX-755 Build all images instead of just Cromwell (broadinstitute#6919) * WX-755 Add `isRelease` option for Docker builds (broadinstitute#6923) * WX-755 Cromwell/CromIAM automatically board train (broadinstitute#6924) * WX-755 Fix environment variable syntax (broadinstitute#6926) * WX-743 Enable TES task creation with BlobPaths (broadinstitute#6921) * Give blob SAS tokens write permission * Case class wrapper for subscription id * Resolve duplicate container name in absolute BlobPath * Ignored test demonstrating correct absolute path generation * Update filesystems/blob/src/test/scala/cromwell/filesystems/blob/BlobPathBuilderSpec.scala Co-authored-by: Brian Reilly <[email protected]> * PR feedback Co-authored-by: Brian Reilly <[email protected]> * [WX-765] Update snakeyaml to 1.33 (broadinstitute#6927) * update snakeyaml to 1.33 * Don't use deprecated no-arg Constructor constructor Co-authored-by: Janet Gainer-Dewar <[email protected]> * WM-1414 Refactoring WesRunLog to omit Cromwell's "workflowLog" object (broadinstitute#6925) * Upgrade Postgres to 42.4.1 (broadinstitute#6932) * WX-735 Fix incorrect and/or nondeterministic filesystem ordering (broadinstitute#6930) * WX-772 Update Scala to 2.13.9 (broadinstitute#6928) * Update Scala to 2.13.9 * Try updating sbt-scoverage * Does this version exist anywhere we can see? * This version actually exists * Update library version to remove conflict * Codegen version * Fix fun new 2.13.9 compiler errors * Resolve warnings * Newest Scala? * I guess not * Does this please Travis? * force ci * Back out changes to generated code Co-authored-by: Adam Nichols <[email protected]> * WX-781 Bump jackson-databind in /CromwellRefdiskManifestCreator (broadinstitute#6935) Bumps [jackson-databind](https://github.com/FasterXML/jackson) from 2.13.2.2 to 2.13.4.1. - [Release notes](https://github.com/FasterXML/jackson/releases) - [Commits](https://github.com/FasterXML/jackson/commits) --- updated-dependencies: - dependency-name: com.fasterxml.jackson.core:jackson-databind dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * WX-808 Host allowlist for HTTP imports (broadinstitute#6938) * `hostAllowlist` that allows everything * Refactor * Stick allow list in HttpResolver * Better default config * Allow list tests * Make it build Co-authored-by: Janet Gainer-Dewar <[email protected]> * Update commons text to 1.10.0 (broadinstitute#6937) * WX-751 Token refresh signal for monitoring (broadinstitute#6939) * Log messages * `DEBUG` -> `INFO` * WX-744 Optionally rewrite blob paths to appear as local paths (broadinstitute#6941) * Modify blob paths for TES * Make blob transformation configurable * Update supportedBackends/tes/src/main/scala/cromwell/backend/impl/tes/TesTask.scala Co-authored-by: Adam Nichols <[email protected]> * Apply PR feedback in second place Co-authored-by: Adam Nichols <[email protected]> * Update changelog for wdl http allow list (broadinstitute#6944) * WM-1491 Fixing Cromwell-client (broadinstitute#6943) * More updated client for use in cbas * Removing excess code * Fix client build script (broadinstitute#6945) * WX-837: Remove CWL references from documentation (broadinstitute#6949) * wx-837 removed cwl references in markdown doc files * wx-837 removed cwlParsingOverview.md, updated mkdocs.yml * wx-837 updated cromwell.yaml, generated new RESTAPI file * WX-728 Add configurable WSM client to Cromwell (broadinstitute#6948) * Dependencies * Compiles but no tests * Formatting * Moar exclusions * Update to latest WSM * Add additional dependency * We need some UUID here to make the request * Formatting * Clarify what is fake * Formatting * Use our own version of Jersey and Jackson stuff * Port-in Khalid's changes (thank you!) Co-authored-by: Khalid Shakir <[email protected]> * Test longevity Don't break the test if someone decides to add a cert to `ws.org` * Cleanup * Cleanup * Cleanup * Adjust TES config file for CI Co-authored-by: Janet Gainer-Dewar <[email protected]> Co-authored-by: Khalid Shakir <[email protected]> * CROM-6554: Removed PAPIv1 references from doc (broadinstitute#6950) * crom-6554 removed references to PAPI v1 from doc * crom-6554 pr feedback, reworded doc to use example conf as a starting point * WX-833 Real Azure DRS Credentials (broadinstitute#6952) * Remove B2C reference from name * Get token for current user rather than getting from KeyVault * Remove KeyVault config for engine * Remove KeyVault config for DRSLocalizer * Remove KeyVault dependency * Remove KeyVault support from localizer repo template * Cleaned up and working Azure token acquisition for engine * Collapse localizer's AccessTokenStrategy into DrsCredentials * Cleanup * WX-853 Remove most CWL (broadinstitute#6955) * WX-696 Enable getting SAS token from WSM (broadinstitute#6954) * WX-696 Enable getting SAS token from WSM * Wire container resource id from config * Move resource-container-id config path * First pass at config for WSM * Remove unused singleton config * Tests for new config * Fix config parsing * Modified b2c token to be provided each time * Remove singletonConfig arg from factory * Restore types to factory configs * Clean up comments and empty token default * Default to config b2c before searching environment * Fix token default on api client * Fix test * Refactor error handling for when there is no token * Remove token constructor arg for clientProvider * Move configs to global singleton config * Update filesystems/blob/src/main/scala/cromwell/filesystems/blob/BlobFileSystemManager.scala * default -> override * Add override token to test * Update filesystems/blob/src/main/scala/cromwell/filesystems/blob/BlobFileSystemManager.scala Co-authored-by: Adam Nichols <[email protected]> * Parentheses * Reduce token timeout * Move AzureCredentials to separate file * Make AzureCredentials an object * WSM token cleanup * Config refactor (broadinstitute#6960) Co-authored-by: Janet Gainer-Dewar <[email protected]> * Initial blob token documentation * Refine language in BlobSasTokenGenerator * Update comment and formatting Co-authored-by: Janet Gainer-Dewar <[email protected]> Co-authored-by: Adam Nichols <[email protected]> * WX-853 Remove CWL language factory, Centaur runner (broadinstitute#6961) * WX-842 Add Pact Dependency for Cromwell (broadinstitute#6962) * WX-842 Add Pact Dependency for Cromwell * Remove incomplete test spec * Initial Pact Test * Fix pact so it compiles * Add breadcrumb comment and clean up * ID-125 Add support for drshub, rename all the things (broadinstitute#6959) * Add support for drshub, rename all the things * fallback to martha if resolver is not in config * WX-867 Translate crc32c hashes to b64 for getm (broadinstitute#6970) * Translate crc32c hashes to b64 for getm * Update tests * Remove obsolete b64 handling for md5, centralize hex validation * Restore old test, fix other test * WX-843 Workflow failure reason should accurately indicate issues opening blob filesystem (broadinstitute#6965) * WX-859 Accept workflow execution identity in config (broadinstitute#6967) * WX-892 Trim down `ValueStore` logging to prevent OOMs (broadinstitute#6981) * Add Nirvana 3.18.1 reference image test, minor cleanup [VS-705] (broadinstitute#6975) * WX-863 Turn off Azure NIO logging (broadinstitute#6982) * Turn off Azure NIO logging * Poke Travis * WM-1616: Allow repeating attempts at initialization (take 2) (broadinstitute#6985) * WX-878 Single shared BlobFileSystemManager (broadinstitute#6986) * Make BlobFileSystemManager shared across all BlobPathBuilders * Update TES conf file to reflect new singleton config * Shell escape reference image files [VS-796] [WX-910] (broadinstitute#6989) * WX-769 `disks` compatibility for TES backend (broadinstitute#6991) * Update FiveMinuteIntro.md (broadinstitute#6994) * WX-906 Sbt Unit Tests as Github Actions (broadinstitute#6992) * WX-926 Support falling back to OCI Manifest Format (broadinstitute#7003) * WX-926 Support falling back to OCI Manifest Forma * Only mount reference disks if requested [WX-925] (broadinstitute#7001) * [WM-1646] Add missing fields for `WorkflowDescription` for WomTool /describe endpoint to Swagger (broadinstitute#7004) * WX-876 Surface TES System Logs to Cromwell when TES backend returns task error status (broadinstitute#6980) * WX-876 Surface TES System Logs to Cromwell when TES backend returns task error status * Address feedback * Address feedback (broadinstitute#6997) * Address additional feedback (broadinstitute#7000) * Fix copy/paste error (broadinstitute#7005) * Address additional feedback * Fix copy/paste error * Trigger CI --------- Co-authored-by: Blair Murri <[email protected]> Co-authored-by: Janet Gainer-Dewar <[email protected]> * Centaur reference image test should validate symlinks [VS-796] (broadinstitute#6996) * WX-903 Pre-GHA test suite disablement * WX-877 Update CHANGELOG for release 85 (broadinstitute#7011) --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Janet Gainer-Dewar <[email protected]> Co-authored-by: Katrina P <[email protected]> Co-authored-by: Chris Llanwarne <[email protected]> Co-authored-by: Christian Freitas <[email protected]> Co-authored-by: Saloni Shah <[email protected]> Co-authored-by: kshakir <[email protected]> Co-authored-by: mspector <[email protected]> Co-authored-by: Adam Nichols <[email protected]> Co-authored-by: Brian Reilly <[email protected]> Co-authored-by: Adam Nichols <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Justin Variath Thomas <[email protected]> Co-authored-by: Christian Freitas <[email protected]> Co-authored-by: Trevyn Langsford <[email protected]> Co-authored-by: Miguel Covarrubias <[email protected]> Co-authored-by: ekiernan <[email protected]> Co-authored-by: Tom Wiseman <[email protected]> Co-authored-by: Blair Murri <[email protected]>
* stuck on globbing * efs works, no callcaching * update readme * extended EFS support * fix for globbing in nested scatters * updated config for globbing, to prevent issues with empty folders
…n of efs-based globs (#32)
* efs fixes : support paths with over 127 characters, fix delocalization of efs-based globs * add deployment manual, fix issue with empty disks * update documentation * update documentation * update documentation
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
deployment documentation, using templates hosted by me
bugfix related to empty disks string
by: @geertvandeweyer