From b1bcefb6a74b0576a6c3b7bceab370647fe0a870 Mon Sep 17 00:00:00 2001 From: jdidion Date: Fri, 24 Mar 2023 14:11:39 -0700 Subject: [PATCH 01/13] update links --- README.md | 24 +++++++++++------------- SPEC.md | 2 +- 2 files changed, 12 insertions(+), 14 deletions(-) diff --git a/README.md b/README.md index 6bba03d8..2d50a70c 100644 --- a/README.md +++ b/README.md @@ -1,9 +1,8 @@ -Workflow Description Language (WDL) -======================================== +# Workflow Description Language (WDL) The **Workflow Description Language (WDL)** is a way to specify data processing workflows with a human-readable and writeable syntax. WDL makes it straightforward to define complex analysis tasks, chain them together in workflows, and parallelize their execution. The language makes common patterns simple to express, while also admitting uncommon or complicated behavior; and strives to achieve portability not only across execution platforms, but also different types of users. Whether one is an analyst, a programmer, an operator of a production system, or any other sort of user, WDL should be accessible and understandable. -# Versioning +## Versioning WDL versioning follows [semantic versioning](https://semver.org) conventions. @@ -11,14 +10,15 @@ The WDL *language* has a two-number version (e.g., 1.1). An increase in the mino The WDL *specification* has a three-number version (e.g. 1.1.1). The specification version tracks the language version, but there may also be patch releases (indicated by a change to the patch, or third, version number) that include fixes for typos, additional examples, or non-breaking clarifications of ambiguous language. -# Language Specifications +## Language Specifications The WDL specification contains all relevant information for users, developers, and engine developers. This GitHub project uses the branch for the current version of the specification as its primary branch, so you will always see the current version of the specification so long as you visit this project's [root URL](https://github.com/openwdl/wdl). -This branch is for version **1.1** of the [WDL language specification](https://github.com/openwdl/wdl/blob/wdl-1.1/SPEC.md). +This branch is for version **1.2** of the [WDL language specification](https://github.com/openwdl/wdl/blob/wdl-1.2/SPEC.md). Previous versions of the spec can be found here: +* [1.1](https://github.com/openwdl/wdl/blob/wdl-1.1/SPEC.md) * [1.0](https://github.com/openwdl/wdl/blob/main/versions/1.0/SPEC.md) There are a number of draft versions (draft 1 - 3) which correspond to our initial efforts at creating WDL. While these are functional specifications, they should not be considered feature complete and contain many bugs and irregularities. Unless absolutely necessary, we would recommend that users should start with the current version of the language. @@ -27,11 +27,9 @@ There are a number of draft versions (draft 1 - 3) which correspond to our initi * [draft-2](https://github.com/openwdl/wdl/blob/main/versions/draft-2/SPEC.md) * [draft-1](https://github.com/openwdl/wdl/blob/main/versions/draft-1/SPEC.md) -The next *minor* version of the specification is [1.2](https://github.com/openwdl/wdl/blob/wdl-1.2/SPEC.md). All development of new *non-breaking* features should be done against that branch. - The next *major* version of the specification is [2.0](https://github.com/openwdl/wdl/blob/wdl-2.0/SPEC.md). All development of new *breaking* features should be done against that branch. -# Community and Support +## Community and Support There are a number of places to ask questions and get involved within the WDL community. Our community thrives the more you get involved and we encourage you to ask questions, provide answers, and make contributions. @@ -41,13 +39,13 @@ There are a number of places to ask questions and get involved within the WDL co - [Support Forum](https://bioinformatics.stackexchange.com/search?q=wdl) - View Previously answered questions about WDL or pose new questions. - [User Guide](https://support.terra.bio/hc/en-us/sections/360007274612-WDL-Documentation) (hosted by the Broad) View a general user guide and simple how-to for WDL -# Published Workflows +## Published Workflows There are many WDL's that have previously been published which provide a good starting point to extend or use as is to fit your workflow needs. While many of these workflows are scattered across the web and in many different repositories, you can find a great selection of high quality, published WDL's available at [Dockstore](https://dockstore.org/search?entryType=workflows&descriptorType=WDL&searchMode=files) as well as a large number of workflows and tasks at [BioWDL](https://github.com/biowdl). Additionally, you can view and test out a number of different workflow's using [Terra](https://app.terra.bio). Please note, that you have to register with Terra in order to view the workflows. -# Software and Tools +## Software and Tools ### Execution Engines @@ -88,16 +86,16 @@ code tests in case the standard simple YAML tests are not sufficient. together with the calling workflow. The zip can be used as an imports zip package for cromwell. The utility can add non-WDL files (such as the license) to the zip package and provides options to package the zip in a binary reproducible way. -# Contributing +## Contributing WDL only advances through community contributions. While submitting an issue is a great way to report a bug in the spec, or create disscussion around current or new features, it will ultimately not translate into an actual change in the spec. The best way to make changes is by submitting a PR. For more information on how you can contribute, please see the [Contributing](CONTRIBUTING.md) readme. Additionally, once a PR has been submitted, it will be subjected to our [RFC Process](RFC.md). -# Governance +## Governance The WDL specification is entirely community driven, however it is overseen by a governance committee. For more information please see the [Governance](GOVERNANCE.md) documentation. -# RFC Process +## RFC Process Any changes submitted to the WDL Specification are subject to the [RFC Process](RFC.md). Please review and familiarize yourself with the process if you would like to see changes submitted to the specification. diff --git a/SPEC.md b/SPEC.md index 549928b3..a56081b1 100644 --- a/SPEC.md +++ b/SPEC.md @@ -1,6 +1,6 @@ # Workflow Description Language (WDL) -This is version 1.1 of the Workflow Description Language (WDL) specification. It introduces a number of new features (denoted by the ✨ symbol) and clarifications to the [1.0](https://github.com/openwdl/wdl/blob/main/versions/1.0/SPEC.md) version of the specification. It also deprecates several aspects of the 1.0 specification that will be removed in the [next major WDL version](https://github.com/openwdl/wdl/blob/wdl-2.0/SPEC.md) (denoted by the 🗑 symbol). +This is version 1.2 of the Workflow Description Language (WDL) specification. It introduces a number of new features (denoted by the ✨ symbol) and clarifications to the [1.1](https://github.com/openwdl/wdl/blob/wdl-1.1/SPEC.md) version of the specification. It also deprecates several aspects of the 1.0 and 1.1 specifications that will be removed in the [next major WDL version](https://github.com/openwdl/wdl/blob/wdl-2.0/SPEC.md) (denoted by the 🗑 symbol). ## Table of Contents From cf597dc07eb88d2fe727bbeb10473569f7b412d8 Mon Sep 17 00:00:00 2001 From: jdidion Date: Fri, 24 Mar 2023 15:38:19 -0700 Subject: [PATCH 02/13] =?UTF-8?q?remove=20=E2=9C=A8?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- SPEC.md | 48 ++++++++++++++++++++++++------------------------ 1 file changed, 24 insertions(+), 24 deletions(-) diff --git a/SPEC.md b/SPEC.md index a56081b1..d2dfa135 100644 --- a/SPEC.md +++ b/SPEC.md @@ -103,8 +103,8 @@ This is version 1.2 of the Workflow Description Language (WDL) specification. It - [Struct Usage](#struct-usage) - [Standard Library](#standard-library) - [Int floor(Float), Int ceil(Float) and Int round(Float)](#int-floorfloat-int-ceilfloat-and-int-roundfloat) - - [✨ Int min(Int, Int), Float min(Float, Float), Float min(Int, Float), Float min(Float, Int)](#-int-minint-int-float-minfloat-float-float-minint-float-float-minfloat-int) - - [✨ Int max(Int, Int), Float max(Float, Float), Float max(Int, Float), Float max(Float, Int)](#-int-maxint-int-float-maxfloat-float-float-maxint-float-float-maxfloat-int) + - [Int min(Int, Int), Float min(Float, Float), Float min(Int, Float), Float min(Float, Int)](#int-minint-int-float-minfloat-float-float-minint-float-float-minfloat-int) + - [Int max(Int, Int), Float max(Float, Float), Float max(Int, Float), Float max(Float, Int)](#int-maxint-int-float-maxfloat-float-float-maxint-float-float-maxfloat-int) - [String sub(String, String, String)](#string-substring-string-string) - [File stdout()](#file-stdout) - [File stderr()](#file-stderr) @@ -132,18 +132,18 @@ This is version 1.2 of the Workflow Description Language (WDL) specification. It - [Array\[Int\] range(Int)](#arrayint-rangeint) - [Array\[Array\[X\]\] transpose(Array\[Array\[X\]\])](#arrayarrayx-transposearrayarrayx) - [Array\[Pair\[X,Y\]\] zip(Array\[X\], Array\[Y\])](#arraypairxy-ziparrayx-arrayy) - - [✨ Pair\[Array\[X\], Array\[Y\]\] unzip(Array\[Pair\[X, Y\]\])](#-pairarrayx-arrayy-unziparraypairx-y) + - [Pair\[Array\[X\], Array\[Y\]\] unzip(Array\[Pair\[X, Y\]\])](#pairarrayx-arrayy-unziparraypairx-y) - [Array\[Pair\[X,Y\]\] cross(Array\[X\], Array\[Y\])](#arraypairxy-crossarrayx-arrayy) - [Array\[X\] flatten(Array\[Array\[X\]\])](#arrayx-flattenarrayarrayx) - [Array\[String\] prefix(String, Array\[P\])](#arraystring-prefixstring-arrayp) - - [✨ Array\[String\] suffix(String, Array\[P\])](#-arraystring-suffixstring-arrayp) - - [✨ Array\[String\] quote(Array\[P\])](#-arraystring-quotearrayp) - - [✨ Array\[String\] squote(Array\[P\])](#-arraystring-squotearrayp) - - [✨ String sep(String, Array\[String\])](#-string-sepstring-arraystring) - - [✨ Array\[Pair\[P, Y\]\] as\_pairs(Map\[P, Y\])](#-arraypairp-y-as_pairsmapp-y) - - [✨ Map\[P, Y\] as\_map(Array\[Pair\[P, Y\]\])](#-mapp-y-as_maparraypairp-y) - - [✨ Array\[P\] keys(Map\[P, Y\])](#-arrayp-keysmapp-y) - - [✨ Map\[P, Array\[Y\]\] collect\_by\_key(Array\[Pair\[P, Y\]\])](#-mapp-arrayy-collect_by_keyarraypairp-y) + - [Array\[String\] suffix(String, Array\[P\])](#arraystring-suffixstring-arrayp) + - [Array\[String\] quote(Array\[P\])](#arraystring-quotearrayp) + - [Array\[String\] squote(Array\[P\])](#arraystring-squotearrayp) + - [String sep(String, Array\[String\])](#string-sepstring-arraystring) + - [Array\[Pair\[P, Y\]\] as\_pairs(Map\[P, Y\])](#arraypairp-y-as_pairsmapp-y) + - [Map\[P, Y\] as\_map(Array\[Pair\[P, Y\]\])](#mapp-y-as_maparraypairp-y) + - [Array\[P\] keys(Map\[P, Y\])](#arrayp-keysmapp-y) + - [Map\[P, Array\[Y\]\] collect\_by\_key(Array\[Pair\[P, Y\]\])](#mapp-arrayy-collect_by_keyarraypairp-y) - [Boolean defined(X?)](#boolean-definedx) - [X select\_first(Array\[X?\]+)](#x-select_firstarrayx) - [Array\[X\] select\_all(Array\[X?\])](#arrayx-select_allarrayx) @@ -567,7 +567,7 @@ Int b = string_to_int["b"] # evaluates to 2 Int c = string_to_int["c"] # error - "c" is not a key in the map ``` -A `Map` is insertion ordered, meaning the order in which elements are added to the `Map` is preserved, for example when [✨ converting a `Map` to an array of `Pair`s](#-arraypairp-y-as_pairsmapp-y). +A `Map` is insertion ordered, meaning the order in which elements are added to the `Map` is preserved, for example when [converting a `Map` to an array of `Pair`s](#-arraypairp-y-as_pairsmapp-y). ```wdl # declaration using a map literal @@ -1214,7 +1214,7 @@ Alternatively, if the command were `"python script.py ~{sep=' ' numbers}"` it wo > 1. `sep` MUST accept only a string as its value > 2. `sep` is only allowed if the type of the expression is `Array[P]` -The `sep` option can be replaced with a call to the ✨ [`sep`](#-string-sepstring-arraystring) function: +The `sep` option can be replaced with a call to the [`sep`](#-string-sepstring-arraystring) function: ```wdl task sep_example { @@ -3536,7 +3536,7 @@ Boolean all_true = [ ] ``` -## ✨ Int min(Int, Int), Float min(Float, Float), Float min(Int, Float), Float min(Float, Int) +## Int min(Int, Int), Float min(Float, Float), Float min(Int, Float), Float min(Float, Int) Returns the smaller of two values. If both values are `Int`s, the return value is an `Int`, otherwise it is a `Float`. @@ -3563,7 +3563,7 @@ workflow min_test { } ``` -## ✨ Int max(Int, Int), Float max(Float, Float), Float max(Int, Float), Float max(Float, Int) +## Int max(Int, Int), Float max(Float, Float), Float max(Int, Float), Float max(Float, Int) Returns the larger of two values. If both values are `Int`s, the return value is an `Int`, otherwise it is a `Float`. @@ -4561,7 +4561,7 @@ Boolean is_true = zipped == [ (1, "a"), (2, "b"), (3, "c") ] Array[Pair[Int, String]] bad = zip(xs, zs) ``` -## ✨ Pair[Array[X], Array[Y]] unzip(Array[Pair[X, Y]]) +## Pair[Array[X], Array[Y]] unzip(Array[Pair[X, Y]]) Creates a `Pair` of `Arrays`, the first containing the elements from the `left` members of an `Array` of `Pair`s, and the second containing the `right` members. This is the inverse of the `zip` function. @@ -4659,7 +4659,7 @@ Array[Array[String]] env3 = [["a", "b], ["c", "d"]] Array[String] bad = prefix("-x ", env3) ``` -## ✨ Array[String] suffix(String, Array[P]) +## Array[String] suffix(String, Array[P]) Adds a suffix to each element of the input array of primitive values. Equivalent to evaluating `"~{array[i]}~{suffix}"` for each `i` in `range(length(array))`. @@ -4684,7 +4684,7 @@ Array[Array[String]] env3 = [["a", "b], ["c", "d"]] Array[String] bad = suffix("-z", env3) ``` -## ✨ Array[String] quote(Array[P]) +## Array[String] quote(Array[P]) Adds double-quotes (`"`) around each element of the input array of primitive values. Equivalent to evaluating `'"~{array[i]}"'` for each `i` in `range(length(array))`. @@ -4704,7 +4704,7 @@ Array[Int] env2 = [1, 2, 3] Array[String] env2_quoted = quote(env2) # ['"1"', '"2"', '"3"'] ``` -## ✨ Array[String] squote(Array[P]) +## Array[String] squote(Array[P]) Adds single-quotes (`'`) around each element of the input array of primitive values. Equivalent to evaluating `"'~{array[i]}'"` for each `i` in `range(length(array))`. @@ -4724,7 +4724,7 @@ Array[Int] env2 = [1, 2, 3] Array[String] env2_quoted = squote(env2) # ["'1'", "'2'", "'3'"] ``` -## ✨ String sep(String, Array[String]) +## String sep(String, Array[String]) Concatenates the elements of an array together into a string with the given separator between consecutive elements. There are always `N-1` separators in the output string, where `N` is the length of the input array. A separator is never added after the last element. @@ -4748,7 +4748,7 @@ Boolean all_true = [ ] ``` -## ✨ Array[Pair[P, Y]] as_pairs(Map[P, Y]) +## Array[Pair[P, Y]] as_pairs(Map[P, Y]) Converts a `Map` with primitive keys into an `Array` of `Pair`s. Since `Map`s are ordered, the output array will always have elements in the same order they were added to the `Map`. @@ -4776,7 +4776,7 @@ workflow foo { } ``` -## ✨ Map[P, Y] as_map(Array[Pair[P, Y]]) +## Map[P, Y] as_map(Array[Pair[P, Y]]) Converts an `Array` of `Pair`s into a `Map` in which the left elements of the `Pair`s are the (primitive) keys and the right elements the values. All the keys must be unique, or an error is raised. The order of the key/value pairs in the output `Map` is the same as the order of the `Pair`s in the `Array`. @@ -4799,7 +4799,7 @@ Boolean is_true2 = as_map(y) == {"a": ("a.bam", "a.bai"), "b": ("b.bam", "b.bai" Boolean bad = as_map([("a", 1), ("a", 2)]) ``` -## ✨ Array[P] keys(Map[P, Y]) +## Array[P] keys(Map[P, Y]) Creates an `Array` of the keys from the input `Map`, in the same order as the elements in the map. @@ -4825,7 +4825,7 @@ workflow foo { } ``` -## ✨ Map[P, Array[Y]] collect_by_key(Array[Pair[P, Y]]) +## Map[P, Array[Y]] collect_by_key(Array[Pair[P, Y]]) Given an `Array` of `Pair`s, creates a `Map` in which the right elements of the `Pair`s are grouped by the left elements. In other words, the input `Array` may have multiple `Pair`s with the same (primitive) key - rather than causing an error (as would happen with [`as_map`](#-mapp-y-as_maparraypairp-y)), all the values with the same key are grouped together into an `Array`. From 1f38247a504e1cbbbfebfcdb359098b4b4c8cf90 Mon Sep 17 00:00:00 2001 From: jdidion Date: Fri, 24 Mar 2023 15:49:32 -0700 Subject: [PATCH 03/13] add contains_key --- SPEC.md | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/SPEC.md b/SPEC.md index d2dfa135..9cd8738d 100644 --- a/SPEC.md +++ b/SPEC.md @@ -143,6 +143,7 @@ This is version 1.2 of the Workflow Description Language (WDL) specification. It - [Array\[Pair\[P, Y\]\] as\_pairs(Map\[P, Y\])](#arraypairp-y-as_pairsmapp-y) - [Map\[P, Y\] as\_map(Array\[Pair\[P, Y\]\])](#mapp-y-as_maparraypairp-y) - [Array\[P\] keys(Map\[P, Y\])](#arrayp-keysmapp-y) + - [✨ Boolean contains\_key(Map\[P, Y\], P), Boolean contains\_key(Map\[P?, Y\], P?)](#-boolean-contains_keymapp-y-p-boolean-contains_keymapp-y-p) - [Map\[P, Array\[Y\]\] collect\_by\_key(Array\[Pair\[P, Y\]\])](#mapp-arrayy-collect_by_keyarraypairp-y) - [Boolean defined(X?)](#boolean-definedx) - [X select\_first(Array\[X?\]+)](#x-select_firstarrayx) @@ -4825,6 +4826,34 @@ workflow foo { } ``` +## ✨ Boolean contains_key(Map[P, Y], P), Boolean contains_key(Map[P?, Y], P?) + +Tests whether the given map contains an entry with the given key. + +**Parameters** + +1. `Map[P, Y]` or `Map[P?, Y]`: `Map` to search for the key. +2. `P` or `P?`: The key to search, of the same type as the `Map`'s key type. If the `Map`'s key type is optional then the key may also be optional. + +**Returns**: `true` if the `Map` contains the key, otherwise false. + +**Example** + +```wdl +version 1.2 + +task get_file { + input { + Map[String?, File] m + String s + } + + output { + File? f = m[s] if contains_key(m, s) else None + } +} +``` + ## Map[P, Array[Y]] collect_by_key(Array[Pair[P, Y]]) Given an `Array` of `Pair`s, creates a `Map` in which the right elements of the `Pair`s are grouped by the left elements. In other words, the input `Array` may have multiple `Pair`s with the same (primitive) key - rather than causing an error (as would happen with [`as_map`](#-mapp-y-as_maparraypairp-y)), all the values with the same key are grouped together into an `Array`. From da3bc652b62b75c69d6e2a5129dcc884f24455b9 Mon Sep 17 00:00:00 2001 From: jdidion Date: Fri, 24 Mar 2023 15:52:12 -0700 Subject: [PATCH 04/13] update changelog --- CHANGELOG.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index c3623b8d..58f34a12 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -42,6 +42,11 @@ version development consist of multiple files. [PR 241](https://github.com/openwdl/wdl/pull/241) by @cjllanwarne. +version 1.1.1 +--------------------------- + ++ Added `contains_key` function to standard library. + version 1.1.0 --------------------------- From bd04046c6c6c108008ca0ae2395799edd4c4cb51 Mon Sep 17 00:00:00 2001 From: jdidion Date: Mon, 27 Mar 2023 12:11:42 -0700 Subject: [PATCH 05/13] allow struct and object arguments to contains_key --- SPEC.md | 99 ++++++++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 81 insertions(+), 18 deletions(-) diff --git a/SPEC.md b/SPEC.md index 9cd8738d..8e9a401c 100644 --- a/SPEC.md +++ b/SPEC.md @@ -143,7 +143,7 @@ This is version 1.2 of the Workflow Description Language (WDL) specification. It - [Array\[Pair\[P, Y\]\] as\_pairs(Map\[P, Y\])](#arraypairp-y-as_pairsmapp-y) - [Map\[P, Y\] as\_map(Array\[Pair\[P, Y\]\])](#mapp-y-as_maparraypairp-y) - [Array\[P\] keys(Map\[P, Y\])](#arrayp-keysmapp-y) - - [✨ Boolean contains\_key(Map\[P, Y\], P), Boolean contains\_key(Map\[P?, Y\], P?)](#-boolean-contains_keymapp-y-p-boolean-contains_keymapp-y-p) + - [✨ Boolean contains\_key(Map\[P, Y\], P), Boolean contains\_key(Struct|Object, String)](#-boolean-contains_keymapp-y-p-boolean-contains_keystructobject-string) - [Map\[P, Array\[Y\]\] collect\_by\_key(Array\[Pair\[P, Y\]\])](#mapp-arrayy-collect_by_keyarraypairp-y) - [Boolean defined(X?)](#boolean-definedx) - [X select\_first(Array\[X?\]+)](#x-select_firstarrayx) @@ -3303,7 +3303,9 @@ workflow foo { A `Struct` type is a user-defined data type. Structs enable the creation of compound data types that bundle together related attributes in a more natural way than is possible using the general-purpose compound types like `Pair` or `Map`. Once defined, a `Struct` type can be used as the type of a declaration like any other data type. -A `struct` definition is a top-level WDL element, meaning it is defined at the same level as tasks and workflows, and it cannot be defined within a task or workflow body. A struct is defined using the `struct` keyword, followed by a name that is unique within the WDL document, and a body containing a set of member declarations. Declarations in a `struct` body differ from those in a `task` or `workflow` in that `struct` members cannot have default initializers. `Struct` members may be optional. +A `struct` definition is a top-level WDL element, meaning it is defined at the same level as tasks and workflows, and it cannot be defined within a task or workflow body. + +A struct is defined using the `struct` keyword, followed by a name that is unique within the WDL document, and a body containing a set of member declarations. Declarations in a `struct` body differ from those in a `task` or `workflow` in that `struct` members cannot have default initializers. `Struct` members may be optional. Valid `struct`: ```wdl @@ -3335,7 +3337,9 @@ struct Sample { ### Struct Literals -A struct literal is an instance of a specific `Struct` type that provides values for all of the non-optional members and any of the optional members. The members of a struct literal are validated against the `Struct`'s definition at the time of creation. Members do not need to be specified in any specific order. Once a struct literal is created, it is immutable like any other WDL value. +A struct literal is an instance of a specific `Struct` type that provides values for all of the non-optional members and any of the optional members. Any optional members that are not initialized are given the value `None`. + +The members of a struct literal are validated against the `Struct`'s definition at the time of creation. Members do not need to be specified in any specific order. Once a struct literal is created, it is immutable like any other WDL value. A struct literal begins with the name of the `Struct` type, followed by name-value pairs for each of the members within braces. @@ -3358,6 +3362,9 @@ task { pin_digits: [1, 2, 3, 4] } + # username is given the value of `None` so we can still access it + Boolean username_is_none = !defined(account1.username) + # error! missing required account_number BankAccount account2 = BankAccount { routing_number: 611325474, @@ -3377,6 +3384,7 @@ task { ``` 🗑 It is also possible to assign an `Object` or `Map[String, X]` value to a `Struct` declaration. In the either case: + * The `Object`/`Map` must not have any members that are not declared for the struct. * The value of each object/map member must be coercible to the declared type of the struct member. * The `Object`/`Map` must at least contain values for all of the struct's non-optional members. @@ -4826,33 +4834,88 @@ workflow foo { } ``` -## ✨ Boolean contains_key(Map[P, Y], P), Boolean contains_key(Map[P?, Y], P?) +## ✨ Boolean contains_key(Map[P, Y], P), Boolean contains_key(Struct|Object, String) + +Given a key-value type collection (`Map`, `Struct`, or `Object`) and a key, tests whether the collection contains an entry with the given key. + +This function has two variants: -Tests whether the given map contains an entry with the given key. +1. `Boolean contains_key(Map[P, Y], P)`: Tests whether the `Map` has an entry with the given key. If `P` is an optional type (e.g., `String?`), then the second argument may be `None`. +2. `Boolean contains_key(Struct|Object, String)`: Tests whether the `Struct` or `Object` has an entry with the given name. In the case of structs and `Object`s, this can be used to test the presence of an optional member, including nested elements. + +For the second variant, the key may specify a nested element of the form "foo.bar" (to any depth). In this case, the first argument is tested for the presence of an element whose key is "foo" and whose value is a key-value type collection (`Map`, `Struct`, or `Object`). If a "foo" element is present, then its value is tested for the presence of an element whose key is "bar". This only tests for the presence of the named element, *not* whether or not it is `defined`. **Parameters** -1. `Map[P, Y]` or `Map[P?, Y]`: `Map` to search for the key. -2. `P` or `P?`: The key to search, of the same type as the `Map`'s key type. If the `Map`'s key type is optional then the key may also be optional. +1. `Map[P, Y]`|`Struct`|`Object`: Collection to search for the key. +2. `P`: The key to search. If the first argument is a `Map`, then the key must be of the same type as the `Map`'s key type. If the `Map`'s key type is optional then the key may also be optional. If The first argument is a `Struct` or `Object`, then the key must be a `String`, and it may specify a nested element, e.g., "foo.bar". -**Returns**: `true` if the `Map` contains the key, otherwise false. +**Returns**: `true` if the collection contains the key, otherwise false. **Example** -```wdl -version 1.2 +
+ + Example: get_values.wdl + + ```wdl + version 1.2 -task get_file { - input { - Map[String?, File] m - String s + struct Person { + String name + Map? details } - output { - File? f = m[s] if contains_key(m, s) else None + workflow get_ints_and_exts { + input { + Map[String, Int] m + String s1 + String s2 + Person p1 + Person p2 + } + + output { + Int? i1 = m[s1] if contains_key(m, s1) else None + Int? i2 = m[s2] if contains_key(m, s2) else None + Int? phone1 = p1.details.phone if contains_key(p1, "details.phone") + Int? phone2 = p2.details.phone if contains_key(p2, "details.phone") + } } -} -``` + ``` + +

+ Example input: + + ```json + { + "get_values.m": {"a": 1, "b": 2}, + "get_values.s1": "a", + "get_values.s2": "c", + "get_values.p1": { + "name": "John", + "details": { + "phone": "123-456-7890" + } + }, + "get_values.p2": { + "name": "Agent X" + } + } + ``` + + Example output: + + ```json + { + "get_ints_and_exts.i1": 1, + "get_ints_and_exts.i2": null, + "get_ints_and_exts.phone1": "123-456-7890", + "get_ints_and_exts.phone2": null, + } + ``` +

+
## Map[P, Array[Y]] collect_by_key(Array[Pair[P, Y]]) From 1af834fa053360f9c84dea734abcfc6bf202bb0c Mon Sep 17 00:00:00 2001 From: jdidion Date: Mon, 27 Mar 2023 13:23:18 -0700 Subject: [PATCH 06/13] make contains_key compound key argument an array rather than string --- CHANGELOG.md | 2 +- SPEC.md | 33 ++++++++++++++++++--------------- 2 files changed, 19 insertions(+), 16 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 58f34a12..c2088fac 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -42,7 +42,7 @@ version development consist of multiple files. [PR 241](https://github.com/openwdl/wdl/pull/241) by @cjllanwarne. -version 1.1.1 +version 1.2.0 --------------------------- + Added `contains_key` function to standard library. diff --git a/SPEC.md b/SPEC.md index 8e9a401c..86564e9d 100644 --- a/SPEC.md +++ b/SPEC.md @@ -143,7 +143,7 @@ This is version 1.2 of the Workflow Description Language (WDL) specification. It - [Array\[Pair\[P, Y\]\] as\_pairs(Map\[P, Y\])](#arraypairp-y-as_pairsmapp-y) - [Map\[P, Y\] as\_map(Array\[Pair\[P, Y\]\])](#mapp-y-as_maparraypairp-y) - [Array\[P\] keys(Map\[P, Y\])](#arrayp-keysmapp-y) - - [✨ Boolean contains\_key(Map\[P, Y\], P), Boolean contains\_key(Struct|Object, String)](#-boolean-contains_keymapp-y-p-boolean-contains_keystructobject-string) + - [✨ Boolean contains\_key(Map\[P, Y\], P), Boolean contains\_key(Object, String), Boolean contains\_key(Map\[String, Y\]|Struct|Object, Array\[String\])](#-boolean-contains_keymapp-y-p-boolean-contains_keyobject-string-boolean-contains_keymapstring-ystructobject-arraystring) - [Map\[P, Array\[Y\]\] collect\_by\_key(Array\[Pair\[P, Y\]\])](#mapp-arrayy-collect_by_keyarraypairp-y) - [Boolean defined(X?)](#boolean-definedx) - [X select\_first(Array\[X?\]+)](#x-select_firstarrayx) @@ -4834,21 +4834,24 @@ workflow foo { } ``` -## ✨ Boolean contains_key(Map[P, Y], P), Boolean contains_key(Struct|Object, String) +## ✨ Boolean contains_key(Map[P, Y], P), Boolean contains_key(Object, String), Boolean contains_key(Map[String, Y]|Struct|Object, Array[String]) Given a key-value type collection (`Map`, `Struct`, or `Object`) and a key, tests whether the collection contains an entry with the given key. -This function has two variants: +This function has thre variants: 1. `Boolean contains_key(Map[P, Y], P)`: Tests whether the `Map` has an entry with the given key. If `P` is an optional type (e.g., `String?`), then the second argument may be `None`. -2. `Boolean contains_key(Struct|Object, String)`: Tests whether the `Struct` or `Object` has an entry with the given name. In the case of structs and `Object`s, this can be used to test the presence of an optional member, including nested elements. +2. `Boolean contains_key(Object, String)`: Tests whether the `Object` has an entry with the given name. +3. `Boolean contains_key(Map[String, Y]|Struct|Object, Array[String])`: Tests recursively for the presence of a compound key within a nested collection. -For the second variant, the key may specify a nested element of the form "foo.bar" (to any depth). In this case, the first argument is tested for the presence of an element whose key is "foo" and whose value is a key-value type collection (`Map`, `Struct`, or `Object`). If a "foo" element is present, then its value is tested for the presence of an element whose key is "bar". This only tests for the presence of the named element, *not* whether or not it is `defined`. +For the third variant, the first argument is a collection that may be nested to any level, i.e., contain values that are collections, which themselves may contain collections, and so on. The second argument is an array of keys that are resolved recursively. If the value associated with any except the last key in the array is `None` or not a collection type, this function returns `false`. + +For example, if the first argument is a `Map[String, Map[String, Int]]` and the second argument is `["foo", "bar"]`, then the outer `Map` is tested for the presence of key "foo", and if it is present, then its value is tested for the presence of key "bar". This only tests for the presence of the named element, *not* whether or not it is `defined`. **Parameters** 1. `Map[P, Y]`|`Struct`|`Object`: Collection to search for the key. -2. `P`: The key to search. If the first argument is a `Map`, then the key must be of the same type as the `Map`'s key type. If the `Map`'s key type is optional then the key may also be optional. If The first argument is a `Struct` or `Object`, then the key must be a `String`, and it may specify a nested element, e.g., "foo.bar". +2. `P|Array[String]`: The key to search. If the first argument is a `Map`, then the key must be of the same type as the `Map`'s key type. If the `Map`'s key type is optional then the key may also be optional. If the first argument is a `Map[String, Y]`, `Struct`, or `Object`, then the key may be either a `String` or `Array[String]`. **Returns**: `true` if the collection contains the key, otherwise false. @@ -4863,23 +4866,23 @@ For the second variant, the key may specify a nested element of the form "foo.ba struct Person { String name - Map? details + Map[String, String]? details } workflow get_ints_and_exts { input { Map[String, Int] m - String s1 - String s2 + String key1 + String key2 Person p1 Person p2 } output { - Int? i1 = m[s1] if contains_key(m, s1) else None - Int? i2 = m[s2] if contains_key(m, s2) else None - Int? phone1 = p1.details.phone if contains_key(p1, "details.phone") - Int? phone2 = p2.details.phone if contains_key(p2, "details.phone") + Int? i1 = m[s1] if contains_key(m, key1) else None + Int? i2 = m[s2] if contains_key(m, key2) else None + String? phone1 = p1.details["phone"] if contains_key(p1, ["details", "phone"]) else None + String? phone2 = p2.details["phone"] if contains_key(p2, ["details", "phone"]) else None } } ``` @@ -4890,8 +4893,8 @@ For the second variant, the key may specify a nested element of the form "foo.ba ```json { "get_values.m": {"a": 1, "b": 2}, - "get_values.s1": "a", - "get_values.s2": "c", + "get_values.key1": "a", + "get_values.key2": "c", "get_values.p1": { "name": "John", "details": { From 5ac10c7199e2407392dc137698a6e00056fc8bd3 Mon Sep 17 00:00:00 2001 From: John Didion Date: Wed, 24 Jan 2024 12:28:50 -0800 Subject: [PATCH 07/13] Update README.md --- tests/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/README.md b/tests/README.md index bbf752d2..037607ff 100644 --- a/tests/README.md +++ b/tests/README.md @@ -1,4 +1,4 @@ Beginning with version 1.1.1, most of the examples in the WDL specification represent test cases. All of these tests conform to the [WDL Markdown Test Specification](https://github.com/openwdl/wdl-tests/docs/MarkdownTests.md). -The tests extracted from the WDL specs can be found in the [wdl-tests](https://github.com/openwdl/wdl-tests/spec/) repository. -This folder contains the [test input files](data/) that are referenced by the test cases in the spec. \ No newline at end of file +The tests extracted from the WDL specs can be found in the [wdl-tests](https://github.com/openwdl/wdl-tests/) repository. +This folder contains the [test input files](data/) that are referenced by the test cases in the spec. From a406975c4c7d3f27e6946e0464334b952213be3b Mon Sep 17 00:00:00 2001 From: John Didion Date: Wed, 24 Jan 2024 12:53:33 -0800 Subject: [PATCH 08/13] Update RFC.md --- RFC.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/RFC.md b/RFC.md index 68bfa24e..f2b00efd 100644 --- a/RFC.md +++ b/RFC.md @@ -3,7 +3,7 @@ RFC Process Most technical decisions are decided through the "RFC" ([Request for Comments](https://en.wikipedia.org/wiki/Request_for_Comments)) process. Small changes, such as minor grammatical edits to the specification, do not need to need to follow the RFC process. However, if one intends to make a substantive change to the WDL specification , the following process should be adhered to: - 1. Ideally have an informal discussion of the topic on the [mailing list](https://groups.google.com/forum/#!forum/openwdl) and/or the [gitter channel](https://gitter.im/openwdl/wdl) in order to gauge basic viability. As a rule of thumb, receiving encouraging feedback from long-standing community members is a good indication that the RFC is worth pursuing. + 1. Ideally have an informal discussion of the topic on the [Slack](https://join.slack.com/t/openwdl/shared_invite/zt-ctmj4mhf-cFBNxIiZYs6SY9HgM9UAVw) and/or [GitHub discussions](https://github.com/openwdl/wdl/discussions) in order to gauge basic viability. As a rule of thumb, receiving encouraging feedback from long-standing community members is a good indication that the RFC is worth pursuing. 2. Write up a formal proposal, including requested changes to the current specification, as a pull request on GitHub 3. A core team member will be assigned as the *shepherd* of this RFC. The shepherd shall be responsible for keeping the discussion moving and ensuring all concerns are responded to. 4. Work to build broad support from the community. Encouraging people to comment, show support, show dissent, etc. Ultimately the level of community support for a change will decide its fate. From 363cbeb1a214db56d1f7b20875af02c5c3aaf8a3 Mon Sep 17 00:00:00 2001 From: John Didion Date: Wed, 24 Jan 2024 12:53:45 -0800 Subject: [PATCH 09/13] Update RFC.md --- RFC.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/RFC.md b/RFC.md index f2b00efd..b49f9723 100644 --- a/RFC.md +++ b/RFC.md @@ -3,7 +3,7 @@ RFC Process Most technical decisions are decided through the "RFC" ([Request for Comments](https://en.wikipedia.org/wiki/Request_for_Comments)) process. Small changes, such as minor grammatical edits to the specification, do not need to need to follow the RFC process. However, if one intends to make a substantive change to the WDL specification , the following process should be adhered to: - 1. Ideally have an informal discussion of the topic on the [Slack](https://join.slack.com/t/openwdl/shared_invite/zt-ctmj4mhf-cFBNxIiZYs6SY9HgM9UAVw) and/or [GitHub discussions](https://github.com/openwdl/wdl/discussions) in order to gauge basic viability. As a rule of thumb, receiving encouraging feedback from long-standing community members is a good indication that the RFC is worth pursuing. + 1. Ideally have an informal discussion of the topic on [Slack](https://join.slack.com/t/openwdl/shared_invite/zt-ctmj4mhf-cFBNxIiZYs6SY9HgM9UAVw) and/or [GitHub discussions](https://github.com/openwdl/wdl/discussions) in order to gauge basic viability. As a rule of thumb, receiving encouraging feedback from long-standing community members is a good indication that the RFC is worth pursuing. 2. Write up a formal proposal, including requested changes to the current specification, as a pull request on GitHub 3. A core team member will be assigned as the *shepherd* of this RFC. The shepherd shall be responsible for keeping the discussion moving and ensuring all concerns are responded to. 4. Work to build broad support from the community. Encouraging people to comment, show support, show dissent, etc. Ultimately the level of community support for a change will decide its fate. From 6107740c9f71b5f5af73448a17e41f4b1e1e1800 Mon Sep 17 00:00:00 2001 From: Patrick Magee Date: Thu, 25 Jan 2024 15:31:12 -0500 Subject: [PATCH 10/13] Update PULL_REQUEST_TEMPLATE.md (#600) * Update PULL_REQUEST_TEMPLATE.md Added a new line referencing prompting the user to add or update examples * Update PULL_REQUEST_TEMPLATE.md updated wording --- .github/PULL_REQUEST_TEMPLATE.md | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 29fde58b..9ba92c3e 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -1,3 +1,4 @@ ### Checklist - [ ] Pull request details were added to CHANGELOG.md +- [ ] Valid examples WDL's were added or updated to the SPEC.md (see the [guide](https://github.com/openwdl/wdl-tests/blob/main/docs/MarkdownTests.md) on writing markdown tests) From e5d1d06bbd99d0760b483ca4191805609e539803 Mon Sep 17 00:00:00 2001 From: Venkat Malladi Date: Tue, 30 Jan 2024 15:59:24 -0600 Subject: [PATCH 11/13] Add back contains_key. --- SPEC.md | 81 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 81 insertions(+) diff --git a/SPEC.md b/SPEC.md index 8931cfcd..4352dca3 100644 --- a/SPEC.md +++ b/SPEC.md @@ -167,6 +167,7 @@ Revisions to this specification are made periodically in order to correct errors - [✨ `as_pairs`](#-as_pairs) - [✨ `as_map`](#-as_map) - [✨ `keys`](#-keys) + - [✨ `contains_key`](#-contains_key) - [✨ `collect_by_key`](#-collect_by_key) - [Other Functions](#other-functions) - [`defined`](#defined) @@ -9209,6 +9210,86 @@ Example output:

+### ✨ `contains_key` + +``` +Map[P, Y], P), Boolean contains_key(Map[P?, Y], P?) +``` + +Given a key-value type collection (`Map`, `Struct`, or `Object`) and a key, tests whether the collection contains an entry with the given key. + +**Parameters** + +1. `Map[P, Y]`|`Struct`|`Object`: Collection to search for the key. +2. `P|Array[String]`: The key to search. If the first argument is a `Map`, then the key must be of the same type as the `Map`'s key type. If the `Map`'s key type is optional then the key may also be optional. If the first argument is a `Map[String, Y]`, `Struct`, or `Object`, then the key may be either a `String` or `Array[String]`. + +**Returns**: `true` if the collection contains the key, otherwise false. + +**Example** + +
+ + Example: get_values.wdl + + ```wdl + version 1.2 + + struct Person { + String name + Map[String, String]? details + } + + workflow get_ints_and_exts { + input { + Map[String, Int] m + String key1 + String key2 + Person p1 + Person p2 + } + + output { + Int? i1 = m[s1] if contains_key(m, key1) else None + Int? i2 = m[s2] if contains_key(m, key2) else None + String? phone1 = p1.details["phone"] if contains_key(p1, ["details", "phone"]) else None + String? phone2 = p2.details["phone"] if contains_key(p2, ["details", "phone"]) else None + } + } + ``` + +

+ Example input: + + ```json + { + "get_values.m": {"a": 1, "b": 2}, + "get_values.key1": "a", + "get_values.key2": "c", + "get_values.p1": { + "name": "John", + "details": { + "phone": "123-456-7890" + } + }, + "get_values.p2": { + "name": "Agent X" + } + } + ``` + + Example output: + + ```json + { + "get_ints_and_exts.i1": 1, + "get_ints_and_exts.i2": null, + "get_ints_and_exts.phone1": "123-456-7890", + "get_ints_and_exts.phone2": null, + } + ``` +

+
+ ### ✨ `collect_by_key` ``` From e6ecb58e8a596cf53bebb5f4675760594e7cb0de Mon Sep 17 00:00:00 2001 From: Venkat Malladi Date: Wed, 31 Jan 2024 09:14:45 -0600 Subject: [PATCH 12/13] Fix bad merge. --- CHANGELOG.md | 2 ++ SPEC.md | 83 +++++----------------------------------------------- 2 files changed, 9 insertions(+), 76 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 8cae58b7..908697c5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -42,7 +42,9 @@ version development consist of multiple files. [PR 241](https://github.com/openwdl/wdl/pull/241) by @cjllanwarne. + + Added `contains_key` function to standard library. + [PR 603](https://github.com/openwdl/wdl/pull/603) version 1.1.1 diff --git a/SPEC.md b/SPEC.md index 4204dd5e..9f03c2fe 100644 --- a/SPEC.md +++ b/SPEC.md @@ -9213,89 +9213,20 @@ Example output: ### ✨ `contains_key` ``` -Map[P, Y], P), Boolean contains_key(Map[P?, Y], P?) +Map[P, Y], Boolean contains_key(Map[P?, Y], P?) ``` Given a key-value type collection (`Map`, `Struct`, or `Object`) and a key, tests whether the collection contains an entry with the given key. -**Parameters** - -1. `Map[P, Y]`|`Struct`|`Object`: Collection to search for the key. -2. `P|Array[String]`: The key to search. If the first argument is a `Map`, then the key must be of the same type as the `Map`'s key type. If the `Map`'s key type is optional then the key may also be optional. If the first argument is a `Map[String, Y]`, `Struct`, or `Object`, then the key may be either a `String` or `Array[String]`. - -**Returns**: `true` if the collection contains the key, otherwise false. - -**Example** - -
- - Example: get_values.wdl - - ```wdl - version 1.2 - - struct Person { - String name - Map[String, String]? details - } - - workflow get_ints_and_exts { - input { - Map[String, Int] m - String key1 - String key2 - Person p1 - Person p2 - } - - output { - Int? i1 = m[s1] if contains_key(m, key1) else None - Int? i2 = m[s2] if contains_key(m, key2) else None - String? phone1 = p1.details["phone"] if contains_key(p1, ["details", "phone"]) else None - String? phone2 = p2.details["phone"] if contains_key(p2, ["details", "phone"]) else None - } - } - ``` - -

- Example input: +This function has thre variants: - ```json - { - "get_values.m": {"a": 1, "b": 2}, - "get_values.key1": "a", - "get_values.key2": "c", - "get_values.p1": { - "name": "John", - "details": { - "phone": "123-456-7890" - } - }, - "get_values.p2": { - "name": "Agent X" - } - } - ``` - - Example output: - - ```json - { - "get_ints_and_exts.i1": 1, - "get_ints_and_exts.i2": null, - "get_ints_and_exts.phone1": "123-456-7890", - "get_ints_and_exts.phone2": null, - } - ``` -

-
+1. `Boolean contains_key(Map[P, Y], P)`: Tests whether the `Map` has an entry with the given key. If `P` is an optional type (e.g., `String?`), then the second argument may be `None`. +2. `Boolean contains_key(Object, String)`: Tests whether the `Object` has an entry with the given name. +3. `Boolean contains_key(Map[String, Y]|Struct|Object, Array[String])`: Tests recursively for the presence of a compound key within a nested collection. +For the third variant, the first argument is a collection that may be nested to any level, i.e., contain values that are collections, which themselves may contain collections, and so on. The second argument is an array of keys that are resolved recursively. If the value associated with any except the last key in the array is `None` or not a collection type, this function returns `false`. -``` -Map[P, Y], P), Boolean contains_key(Map[P?, Y], P?) -``` - -Given a key-value type collection (`Map`, `Struct`, or `Object`) and a key, tests whether the collection contains an entry with the given key. +For example, if the first argument is a `Map[String, Map[String, Int]]` and the second argument is `["foo", "bar"]`, then the outer `Map` is tested for the presence of key "foo", and if it is present, then its value is tested for the presence of key "bar". This only tests for the presence of the named element, *not* whether or not it is `defined`. **Parameters** From e6f654ddbd2b85b949a5b36413a6fa1539ece718 Mon Sep 17 00:00:00 2001 From: Venkat Malladi Date: Wed, 31 Jan 2024 12:26:44 -0600 Subject: [PATCH 13/13] Fix codeblock. --- SPEC.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/SPEC.md b/SPEC.md index 9f03c2fe..ee05dc3a 100644 --- a/SPEC.md +++ b/SPEC.md @@ -9213,7 +9213,9 @@ Example output: ### ✨ `contains_key` ``` -Map[P, Y], Boolean contains_key(Map[P?, Y], P?) +* Boolean contains_key(Map[P, Y], P) +* Boolean contains_key(Object, String) +* Boolean contains_key(Map[String, Y]|Struct|Object, Array[String]) ``` Given a key-value type collection (`Map`, `Struct`, or `Object`) and a key, tests whether the collection contains an entry with the given key.