Add select parameter with options from remote resources #17087

mvdbeek · 2023-11-27T13:16:27Z

Updating regular tool data tables for frequently changing data is a little tricky, and it was requested that we allow loading options dynamically from a URL. The eventual goals of this work is to enable exploring resources with great APIs (NCBI datasets is a good example) from within Galaxy tools.

Includes a little mini-language for manipulating arbitrary JSON responses using ECMAScript expressions.

From the schema:

`from_url`

The following example demonstrates getting options from a third-party server
with server side requests.

<param name="url_param_value" type="select">
    <options from_url="https://usegalaxy.org/api/genomes">
    </options>
</param>

Here a GET request is made to https://usegalaxy.org/api/genomes, which returns
an array of arrays, such as

[
    ["unspecified (?)", "?"],
    ["A. ceylanicum Mar. 2014 (WS243/Acey_2013.11.30.genDNA/ancCey1) (ancCey1)", "ancCey1"],
    ...
]

Each inner array is a user-selectable option, where the first item in the inner array
is the name of the option (as shown in the select field in the user interface), and
the second option is the value that is passed on to the tool. An optional third
element can be added to the inner array which corresponds to the selected state.
If the third item is true then this particular option is pre-selected.

A more complicated example is shown below, where a POST request is made with a templated
request header and body. The upstream response is then also transformed using an ecma 5.1
expression:

<param name="url_param_value_header_and_body" type="select">
    <options from_url="https://postman-echo.com/post" request_method="POST">
        <!-- Example for accessing user secrets via extra preferences -->
        <request_headers type="json">
            {"x-api-key": "${__user__.extra_preferences.fake_api_key if $__user__ else "anon"}"}
        </request_headers>
        <request_body type="json">
            {"name": "value"}
        </request_body>
        <!-- https://postman-echo.com/post echos values sent to it, so here we list the response headers -->
        <postprocess_expression type="ecma5.1"><![CDATA[${
            return Object.keys(inputs.headers).map((header) => [header, header])
        }]]></postprocess_expression>
    </options>
</param>

The header and body templating mechanism can be used to access protected resources,
and the postprocess_expression can be used to transform arbitrary JSON responses
to arrays of name and value, or arrays of name, value and selected.

For an example tool see select_from_url.xml.

How to test the changes?

(Select all options that apply)

I've included appropriate automated tests.
This is a refactoring of components with existing test coverage.
Instructions for manual testing are as follows:
1. [add testing steps and prerequisites here if you didn't write automated tests covering all your changes]

License

I agree to license these and all my past contributions to the core galaxy codebase under the MIT license.

jmchilton · 2023-11-27T15:34:54Z

Nifty!

bernt-matthias · 2024-01-29T08:16:00Z

lib/galaxy/tool_util/xsd/galaxy.xsd

@@ -4442,6 +4448,7 @@ used to generate dynamic options.
      <xs:element name="filter" type="Filter" minOccurs="0" maxOccurs="unbounded"/>
      <xs:element name="column" type="Column" minOccurs="0" maxOccurs="unbounded"/>
      <xs:element name="validator" type="Validator" minOccurs="0" maxOccurs="1"/>
+      <xs:element name="postprocess_expression" type="Expression" minOccurs="0" maxOccurs="1"/>


Maybe link the test tool from here?

Maybe a linter checking that postprocess_expression only appears for from_url could be nice. But this would only make sense with the big linter PR merged.

test/functional/tools/select_from_url.xml

bernt-matthias · 2024-01-29T09:45:12Z

lib/galaxy/tools/parameters/dynamic_options.py

@@ -766,6 +774,39 @@ def get_field_by_name_for_value(self, field_name, value, trans, other_values):
        return rval

    def get_options(self, trans, other_values):
+        def to_triple(values):
+            if len(values) == 2:


Should we cover empty values or values of length 1? Or non-dict values?

Do you have an example for when that would be useful ? Keep in mind that the expression is evaluated on the request's response

We don't know what the response is, or? Could be just a list of length 1, an empty list, or a string.

The point is that you can transform anything that is returned, so IMO that is where the complexity should go, not here.

Thanks for the explanation. That's fine with me. Could you add this to the docs for from_url, i.e. that the code expects an array of length 2 or 3 in the response, or the response being transformed.

Yes, definitely, there's some expansion needed.

Updating regular tool data tables for frequently changing data is a little tricky, and it was requested that we allow loading options dynamically from a URL. The eventual goals of this work is to enable exploring resources with great APIs (NCBI datasets is a good exmaple) from within Galaxy tools. This doesn't do any of that yet ... it's just a simple URL one can query with a GET request, and the data there must be in the format ["name", "value", True|False]. Following steps: - [x] allow templating (Cheetah ? JS?) - [x] allow post-processing to extract name-value pairs - jq-like syntax ? - [x] JS expression ? - add pagination (requires work on vue-multiselect) - add request caching Some potential issues: - Slow server responses: use async library and cancel request after N seconds ? - Large respons ... stream response and cancel after configurable max response size ?

Context are the user context values, we could extend that as needed.

…xpression

mvdbeek · 2024-02-14T15:06:49Z

@bernt-matthias what do you think, I've added some documentation to the xsd (also in the PR description)

bernt-matthias

Excellent :)

github-actions bot added area/testing area/workflows area/tool-framework labels Nov 27, 2023

mvdbeek changed the title ~~Add <options from_url="https://example.com"></options>~~ Add select parameter with options from remote resources Nov 27, 2023

jmchilton approved these changes Nov 27, 2023

View reviewed changes

mvdbeek force-pushed the from_url_option branch from 2bf3ac6 to 3398c83 Compare November 28, 2023 16:30

mvdbeek marked this pull request as ready for review November 28, 2023 16:50

github-actions bot added this to the 23.2 milestone Nov 28, 2023

mvdbeek force-pushed the from_url_option branch 2 times, most recently from 86545e5 to 5390ef1 Compare November 29, 2023 08:46

mvdbeek modified the milestones: 23.2, 24.0 Dec 19, 2023

bernt-matthias reviewed Jan 29, 2024

View reviewed changes

mvdbeek added 8 commits February 12, 2024 15:47

Allow templating URL with cheetah

5eecab4

Context are the user context values, we could extend that as needed.

Implement postprocessing using javascript sandbox expression evaluation

6fc51bd

Create simplified do_eval wrapper

1b7fc8a

Add test cases against usegalaxy.org's genomes api

d96dc3d

Fix tool schema for postprocess_expression

648b29b

Allow expression evaluation to fail, so default value can be set in e…

2eca43e

…xpression

Cache requests across trans session

2f29300

mvdbeek force-pushed the from_url_option branch from 5390ef1 to f0bd398 Compare February 12, 2024 18:25

bernt-matthias mentioned this pull request Feb 12, 2024

Omero upload Helmholtz-UFZ/galaxy-tools#17

Closed

mvdbeek force-pushed the from_url_option branch 2 times, most recently from aa87988 to 204dee1 Compare February 13, 2024 11:31

mvdbeek added 2 commits February 13, 2024 14:51

Implement request header and body templating

5aabab9

Add documentation for from_url options

e02dd32

mvdbeek force-pushed the from_url_option branch from 204dee1 to e02dd32 Compare February 13, 2024 13:51

mvdbeek requested a review from a team February 13, 2024 14:10

bernt-matthias approved these changes Feb 14, 2024

View reviewed changes

mvdbeek merged commit ddabba7 into galaxyproject:dev Feb 14, 2024
53 checks passed

nsoranzo deleted the from_url_option branch February 14, 2024 17:53

nsoranzo added the kind/enhancement label Feb 14, 2024

galaxyproject deleted a comment from github-actions bot Feb 14, 2024

hexylena mentioned this pull request Feb 26, 2024

Replace (some of?) webapollo.py's interactions with Apollo, with new Galaxy feature galaxy-genome-annotation/galaxy-tools#93

Open

bernt-matthias mentioned this pull request Dec 10, 2024

[24.0] Linter: allow dynamic option definition by from_url #19300

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add select parameter with options from remote resources #17087

Add select parameter with options from remote resources #17087

mvdbeek commented Nov 27, 2023 •

edited

Loading

jmchilton commented Nov 27, 2023

bernt-matthias Jan 29, 2024

bernt-matthias Jan 29, 2024

mvdbeek Jan 29, 2024

bernt-matthias Jan 29, 2024

mvdbeek Jan 29, 2024

bernt-matthias Jan 29, 2024

mvdbeek Jan 29, 2024

mvdbeek commented Feb 14, 2024

bernt-matthias left a comment

Add select parameter with options from remote resources #17087

Add select parameter with options from remote resources #17087

Conversation

mvdbeek commented Nov 27, 2023 • edited Loading

from_url

How to test the changes?

License

jmchilton commented Nov 27, 2023

bernt-matthias Jan 29, 2024

Choose a reason for hiding this comment

bernt-matthias Jan 29, 2024

Choose a reason for hiding this comment

mvdbeek Jan 29, 2024

Choose a reason for hiding this comment

bernt-matthias Jan 29, 2024

Choose a reason for hiding this comment

mvdbeek Jan 29, 2024

Choose a reason for hiding this comment

bernt-matthias Jan 29, 2024

Choose a reason for hiding this comment

mvdbeek Jan 29, 2024

Choose a reason for hiding this comment

mvdbeek commented Feb 14, 2024

bernt-matthias left a comment

Choose a reason for hiding this comment

mvdbeek commented Nov 27, 2023 •

edited

Loading

`from_url`