Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update tpv on production to version 2.2.0 #1158

Merged
merged 23 commits into from
Mar 26, 2023

Conversation

cat-bro
Copy link
Collaborator

@cat-bro cat-bro commented Jan 30, 2023

Update tpv on production to 2.1.0.

Job destinations are defined in tpv destinations.yml.j2 instead of job conf.
Job conf limits are now in aarnet.usegalaxy.org.au and templated into the job conf.
Pull request linting has updated to check templating of files as well as tpv lint.
TPV shared rules for tools will be used in addition to Galaxy Australia's tool rules. GA's rules take precedence.
GA tool rules now always have mem values and rule IDs.
This PR also removes some legacy dtd code.

I'm changing this from 'draft' to 'ready for review' but it can't be merged without first disabling the automatic copying of tpv files to production. I'd like to merge this early in the day when there is a lower rate of job submission.

@cat-bro cat-bro force-pushed the production-tpv-upgrade branch 2 times, most recently from 30769ce to f86b425 Compare March 17, 2023 05:54
@cat-bro cat-bro force-pushed the production-tpv-upgrade branch 2 times, most recently from 1db47bc to aea121a Compare March 21, 2023 02:43
@cat-bro cat-bro force-pushed the production-tpv-upgrade branch from 2cf0278 to fad9849 Compare March 21, 2023 05:53
cat-bro added 2 commits March 21, 2023 17:17
…o that highest non-value spec is default and everything else is small_input_rule, medium_input_rule, fail_rule
@cat-bro cat-bro force-pushed the production-tpv-upgrade branch from fad9849 to c5dbf02 Compare March 22, 2023 00:55
@cat-bro
Copy link
Collaborator Author

cat-bro commented Mar 22, 2023

Tool specifications in tools.yml have previously had lowest cores/mem as default and higher resource allocations set in conditional rules. This changes to have the non-failing condition with the highest cores/mem as default and lower resource allocations set in conditional rules, e.g. deeptools_plot_heatmap

  toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_plot_heatmap/deeptools_plot_heatmap/.*:
    cores: 3
    rules:
    - if: input_size >= 0.3
      cores: 7

changes to

  toolshed.g2.bx.psu.edu/repos/bgruening/deeptools_plot_heatmap/deeptools_plot_heatmap/.*:
    cores: 7
    mem: 26.9
    rules:
    - id: deeptools_plot_heatmap_small_input_rule
      if: input_size < 0.3
      cores: 3
      mem: 11.5

Using this convention should make it easier to switch to using a shared database rule for a tool. If we wanted to use shared_db for deeptools_plot_heatmap we could either delete the entry entirely or delete {cores: 7, mem: 26.9} from the above and keep using the small input rule.

@cat-bro cat-bro changed the title Production tpv upgrade [DO NOT MERGE] Update tpv on production to version 2.1.0 Mar 22, 2023
@cat-bro cat-bro marked this pull request as ready for review March 22, 2023 02:22
@cat-bro
Copy link
Collaborator Author

cat-bro commented Mar 22, 2023

This PR doesn't change the rank function in any way but I do think it needs to change. I'm working on alternatives to the rank function on dev. We can swap the rank function later on when we have seen what the new edge cases will be on production.

@cat-bro cat-bro requested a review from nuwang March 22, 2023 02:24
.github/workflows/check_tpv_config.yml Show resolved Hide resolved
)

tag_values_match(entity, include_tags, exclude_tags)
# helpers.tag_values_match(entity, include_tags, exclude_tags)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can make a release of tpv immediately so you can use this instead?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That would be good!

@cat-bro cat-bro changed the title [DO NOT MERGE] Update tpv on production to version 2.1.0 [DO NOT MERGE] Update tpv on production to version 2.2.0 Mar 22, 2023
nuwang
nuwang previously approved these changes Mar 22, 2023
Copy link
Contributor

@nuwang nuwang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@cat-bro cat-bro changed the title [DO NOT MERGE] Update tpv on production to version 2.2.0 Update tpv on production to version 2.2.0 Mar 26, 2023
@cat-bro cat-bro merged commit 6eb2f7a into usegalaxy-au:master Mar 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants