Update the scripts used to generate the ucsc-XYZ recipes #50325

martin-g · 2024-08-26T11:46:54Z

This PR updates the scripts used to generate the ucsc-xyz recipes.
See #50323 and #50324 for examples of the changes in the recipes.

Once this PR is merged the other recipes may be pushed in bulk ?!

Please read the guidelines for Bioconda recipes before opening a pull request (PR).

General instructions

If this PR adds or updates a recipe, use "Add" or "Update" appropriately as the first word in its title.
New recipes not directly relevant to the biological sciences need to be submitted to the conda-forge channel instead of Bioconda.
PRs require reviews prior to being merged. Once your PR is passing tests and ready to be merged, please issue the @BiocondaBot please add label command.
Please post questions on Gitter or ping @bioconda/core in a comment.

Instructions for avoiding API, ABI, and CLI breakage issues

Conda is able to record and lock (a.k.a. pin) dependency versions used at build time of other recipes.
This way, one can avoid that expectations of a downstream recipe with regards to API, ABI, or CLI are violated by later changes in the recipe.
If not already present in the meta.yaml, make sure to specify run_exports (see here for the rationale and comprehensive explanation).
Add a run_exports section like this:

build:
  run_exports:
    - ...

with ... being one of:

Case	run_exports statement
semantic versioning	`{{ pin_subpackage("myrecipe", max_pin="x") }}`
semantic versioning (0.x.x)	`{{ pin_subpackage("myrecipe", max_pin="x.x") }}`
known breakage in minor versions	`{{ pin_subpackage("myrecipe", max_pin="x.x") }}` (in such a case, please add a note that shortly mentions your evidence for that)
known breakage in patch versions	`{{ pin_subpackage("myrecipe", max_pin="x.x.x") }}` (in such a case, please add a note that shortly mentions your evidence for that)
calendar versioning	`{{ pin_subpackage("myrecipe", max_pin=None) }}`

while replacing "myrecipe" with either name if a name|lower variable is defined in your recipe or with the lowercase name of the package in quotes.

Bot commands for PR management

Please use the following BiocondaBot commands:

Everyone has access to the following BiocondaBot commands, which can be given in a comment:

`@BiocondaBot please update`	Merge the master branch into a PR.
`@BiocondaBot please add label`	Add the `please review & merge` label.
`@BiocondaBot please fetch artifacts`	Post links to CI-built packages/containers. You can use this to test packages locally.

Note that the @BiocondaBot please merge command is now depreciated. Please just squash and merge instead.

Also, the bot watches for comments from non-members that include @bioconda/<team> and will automatically re-post them to notify the addressed <team>.

martin-g · 2024-08-26T11:50:54Z

@bgruening This is the result of your suggestion at #50106 (comment)

What would be the best way to rebuild the recipes ?

should I open PRs against master in mini-bulks (i.e. 5-6 recipes in one PR) ?
should someone (@aliciaaevans ?!) do it in the bulk branch ? (I don't know how exactly the bulk branch changes work...)

aliciaaevans · 2024-08-26T12:41:32Z

Oh cool, I didn't realize these scripts existed. I could run this on bulk once it's merged. Should be relatively quick since these aren't interdependent like bioconductor recipes.

bgruening

Awesome, thanks a lot!

bgruening · 2024-08-26T12:11:35Z

scripts/ucsc/template-meta-with-python.yaml

@@ -23,14 +26,14 @@ requirements:
    - libpng
    - libuuid
    - mysql-connector-c
-    - openssl
+    - libopenssl-static


interesting why is this needed?

The tools require libssl.a instead of libssl.so. I am not sure since when though.
I have seen some recipes using includes.patch to change .a to .so. I decided to keep the original build, i.e. to not patch it, and use the new dependency instead.

scripts/ucsc/template-meta-with-python.yaml

bgruening · 2024-08-26T12:48:05Z

scripts/ucsc/template-meta-with-python.yaml

+  doc_url: "https://github.com/ucscGenomeBrowser/kent/blob/master/README"
+
+extra:
+  additional-platforms:


Should we add a doi as reference here: https://genome.ucsc.edu/cite.html#ref

Maybe the bigwig, bigbed paper? https://doi.org/10.1093/bioinformatics/btq351

Could you please give me the YAML snippet that I should add? Or please push it yourself to this PR!

bgruening · 2024-08-26T12:48:56Z

scripts/ucsc/template-build-cp.sh

 mkdir -p "$PREFIX/bin"
-cp {program_source_dir}/{program} "$PREFIX/bin"
-chmod +x "$PREFIX/bin/{program}"
+cp -f {program_source_dir}/{program} "${{PREFIX}}/bin"


Is the force needed, or are we masking potential error here?

This is an improvement by @mencian to one of my previous PRs.
I could remove it if there is a chance to mask some problem!

I'm not sure about the reason, so I would rather remove.

bgruening · 2024-08-26T12:50:46Z

This is super cool, thanks for updating them. We should use the bulk branch here, that why we have it. Mini-bulks are just wasting your time.

martin-g · 2024-08-26T12:52:12Z

Thanks, @aliciaaevans !

There are ~200 disabled ucsc- recipes in build-fail-blacklist that should be re-enabled before running the bulk build!
There will be (hopefully few) recipes like #49297 which will fail on linux-aarch64 - for them we will need to remove the additional-platforms entry. Actually I could add a new array in the Python script for the known ones, so they are skipped automatically. Let me update the PR!

martin-g · 2024-08-26T13:41:05Z

Actually I could add a new array in the Python script for the known ones, so they are skipped automatically. Let me update the PR!

Done!

bgruening · 2024-08-27T06:22:16Z

scripts/ucsc/template-build-cp.sh

 mkdir -p "$PREFIX/bin"
-cp {program_source_dir}/{program} "$PREFIX/bin"
-chmod +x "$PREFIX/bin/{program}"
+cp -f {program_source_dir}/{program} "${{PREFIX}}/bin"


I'm not sure about the reason, so I would rather remove.

scripts/ucsc/template-meta.yaml

bioconda#50325 (review) Signed-off-by: Martin Tzvetanov Grigorov <[email protected]>

bgruening

Found a better one, sorry.

scripts/ucsc/template-meta-with-python.yaml

scripts/ucsc/template-meta.yaml

Signed-off-by: Martin Tzvetanov Grigorov <[email protected]>

…h known build issues on aarch64 Signed-off-by: Martin Tzvetanov Grigorov <[email protected]>

Co-authored-by: Björn Grüning <[email protected]>

bioconda#50325 (review) Signed-off-by: Martin Tzvetanov Grigorov <[email protected]>

Signed-off-by: Martin Tzvetanov Grigorov <[email protected]>

Co-authored-by: Björn Grüning <[email protected]>

Set -x for easier debugging Signed-off-by: Martin Tzvetanov Grigorov <[email protected]>

martin-g · 2024-08-27T07:53:00Z

@BiocondaBot please add label

To be consistent with the SKIP list Signed-off-by: Martin Tzvetanov Grigorov <[email protected]>

bgruening

Thanks a lot @martin-g

bgruening · 2024-08-27T11:49:16Z

scripts/ucsc/create-ucsc-packages.py

@@ -198,6 +197,12 @@ def program_subdir(program, names):
    'gfServer',
 ]

+# A list of programs which have problems to build on linux-aarch64
+SKIP_AARCH64 = [


I think at some point we should have a global list, that script like this one can consume.

aliciaaevans · 2024-08-27T14:33:43Z

@martin-g I've got time today to get this started on bulk.

aliciaaevans · 2024-08-27T19:01:58Z

Still working though timeouts and disk space limits, but here are failures so far: https://github.com/bioconda/bioconda-recipes/wiki/build-failures-bulk

I think the aarch failures are all that knet_init_alt so we can remove the platform for those after they've all been built or failed. I think the linux-64 are mostly one error: multiple definition of htmlRecover.

aliciaaevans · 2024-08-27T22:28:43Z

(I'm still using this PR for notes about this bulk update for ucsc.)

I've removed linux-aarch64 from the failures with the knet_init_alt error, removed the build_failure files, and added them to the SKIP_AARCH64 in the python script (all on the bulk branch). Now https://github.com/bioconda/bioconda-recipes/wiki/build-failures-bulk only has other failures. I will probably merge bulk back to master sooner rather than later (tomorrow?). We can always do another bulk run to fix the knet_init_alt issue and the remaining failures can be individual PRs.

martin-g · 2024-08-28T07:23:58Z

Thank you, @aliciaaevans !

In #50368 @mencian added support for osx-64 to one of the ucsc recipes.
After running the scripts in bulk this change will be overwritten and lost.
@mencian We can add the OSX specific lines to the templates once @aliciaaevans merges the current improvements back to master.

This was referenced Aug 26, 2024

ucsc-pslmrnacover: add linux-aarch64 build #50324

Merged

Update ucsc-addCols to v469 #50323

Merged

bgruening reviewed Aug 26, 2024

View reviewed changes

bgruening reviewed Aug 27, 2024

View reviewed changes

martin-g added a commit to martin-g/bioconda-recipes that referenced this pull request Aug 27, 2024

Do not use -f (force) when copying the binaries to $PREFIX/bin

7944e4a

bioconda#50325 (review) Signed-off-by: Martin Tzvetanov Grigorov <[email protected]>

bgruening reviewed Aug 27, 2024

View reviewed changes

scripts/ucsc/template-meta-with-python.yaml Outdated Show resolved Hide resolved

scripts/ucsc/template-meta.yaml Outdated Show resolved Hide resolved

martin-g and others added 6 commits August 27, 2024 10:24

Update the scripts used to generate the ucsc-XYZ recipes

a5d17eb

Signed-off-by: Martin Tzvetanov Grigorov <[email protected]>

Do not render linux-aarch64 as an additional platform for recipes wit…

8c01464

…h known build issues on aarch64 Signed-off-by: Martin Tzvetanov Grigorov <[email protected]>

Add identifiers

bd8b813

Co-authored-by: Björn Grüning <[email protected]>

Do not use -f (force) when copying the binaries to $PREFIX/bin

eedaf51

bioconda#50325 (review) Signed-off-by: Martin Tzvetanov Grigorov <[email protected]>

Move extra > identifiers above the additional-platforms

b3f2cf2

Signed-off-by: Martin Tzvetanov Grigorov <[email protected]>

Update doi identifier

12d77ef

Co-authored-by: Björn Grüning <[email protected]>

martin-g force-pushed the update-ucsc-recipe-generator branch from 19753d4 to 12d77ef Compare August 27, 2024 07:24

Ignore template-meta.yaml when checking for additional_platforms

8275e42

Set -x for easier debugging Signed-off-by: Martin Tzvetanov Grigorov <[email protected]>

BiocondaBot added the please review & merge set to ask for merge label Aug 27, 2024

martin-g and others added 2 commits August 27, 2024 10:54

Rename NO_AARCH64 list to SKIP_AARCH64

46082c2

To be consistent with the SKIP list Signed-off-by: Martin Tzvetanov Grigorov <[email protected]>

Apply suggestions from code review

171d333

bgruening approved these changes Aug 27, 2024

View reviewed changes

aliciaaevans merged commit f5257d5 into bioconda:master Aug 27, 2024
6 checks passed

martin-g deleted the update-ucsc-recipe-generator branch August 28, 2024 07:21

This was referenced Aug 28, 2024

ci: test improvements to additional platform script #50388

Draft

ci: Fix platform script echo #50389

Merged

Update pytrimal to 0.8.0 #50383

Merged

Bulk: UCSC updates #50393

Merged

Bump ucsc tools #44126

Closed

Update UCSC recipes #37618

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update the scripts used to generate the ucsc-XYZ recipes #50325

Update the scripts used to generate the ucsc-XYZ recipes #50325

martin-g commented Aug 26, 2024

martin-g commented Aug 26, 2024 •

edited

Loading

aliciaaevans commented Aug 26, 2024

bgruening left a comment

bgruening Aug 26, 2024

martin-g Aug 26, 2024

bgruening Aug 26, 2024

martin-g Aug 26, 2024

bgruening Aug 26, 2024

martin-g Aug 26, 2024

bgruening Aug 27, 2024

bgruening commented Aug 26, 2024

martin-g commented Aug 26, 2024

martin-g commented Aug 26, 2024

bgruening Aug 27, 2024

bgruening left a comment

martin-g commented Aug 27, 2024

bgruening left a comment

bgruening Aug 27, 2024

aliciaaevans commented Aug 27, 2024

aliciaaevans commented Aug 27, 2024

aliciaaevans commented Aug 27, 2024

martin-g commented Aug 28, 2024

Update the scripts used to generate the ucsc-XYZ recipes #50325

Update the scripts used to generate the ucsc-XYZ recipes #50325

Conversation

martin-g commented Aug 26, 2024

General instructions

Instructions for avoiding API, ABI, and CLI breakage issues

Bot commands for PR management

martin-g commented Aug 26, 2024 • edited Loading

aliciaaevans commented Aug 26, 2024

bgruening left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bgruening commented Aug 26, 2024

martin-g commented Aug 26, 2024

martin-g commented Aug 26, 2024

Choose a reason for hiding this comment

bgruening left a comment

Choose a reason for hiding this comment

martin-g commented Aug 27, 2024

bgruening left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aliciaaevans commented Aug 27, 2024

aliciaaevans commented Aug 27, 2024

aliciaaevans commented Aug 27, 2024

martin-g commented Aug 28, 2024

martin-g commented Aug 26, 2024 •

edited

Loading