Automatically pushing of repos to MTS

[hexylena]

According to @martenson, the MTS is now running 15.05, so I'd like to open discussion as to pushing to the MTS.

• Updates to recursive uploads planemo#71
• Utility addition to recursive uploads allowing filtering based on git diffs planemo#225

Points for discussion

• What bugs do we need to solve before we can turn that on?
  • Is this creating extraneous installable revisions?
  • Is this polluting the TS?
  • Do I need to implement Updates to recursive uploads planemo#71 to automatically determine proper push order, or is simply pushing everything in the git repo a couple times "good enough"?
• Is everyone OK with automating the push process?

cc @bgruening @peterjc @nsoranzo

[jmchilton]

As a precursor to this - can you add --check_diff as part of shed_upload that you do for the test tool shed? This should prevent the creation of extraneous install-able revisions.

[martenson]

@jmchilton nice feature!

[hexylena]

for instance you can imagine automatically pushing everything committed to github to the test tool shed and if the tests pass there with shed_test then promoting them to the main tool shed

This will be a thing soon, and I'll make sure it's part of the upload process. If the above functions, would all of the IUC be okay with automated pushing to MTS?

[hexylena]

Y'all, I've had two requests in the past couple of days for packages where someone tracked me down as the last committer on a package wondering why it wasn't on the MTS.

Running a quick test of what's in the MTS vs our github...It would appear that we're missing a non-insignificant number of packages on the MTS: a full 68/194 packages weren't on the MTS. I'm pushing those manually now.

[bgruening]

@erasche let me now the list of packages and I can help

[hexylena]

@bgruening when I said "by hand" I mean "in a for loop" but thanks :) Thankfully since the TS is running 15.05 we can use the --force_repository_creation and it all runs very smoothly.

There were some bugs which need fixing though! @jmchilton can we consider more adding description in .shed.yml to galaxyproject/planemo#107 ?

cd '/home/users/cpt/cpt/esr/Projects/galaxy/tools-iuc/packages/package_boost_1_57' && git rev-parse HEAD
cd '/home/users/cpt/cpt/esr/Projects/galaxy/tools-iuc/packages/package_boost_1_57' && git diff --quiet
description required for automatic creation of repositories

cd '/home/users/cpt/cpt/esr/Projects/galaxy/tools-iuc/packages/package_homer_4_2' && git rev-parse HEAD
cd '/home/users/cpt/cpt/esr/Projects/galaxy/tools-iuc/packages/package_homer_4_2' && git diff --quiet
Could not update package_homer_4_2
Unable to locate repository with name package_weblogo_2_8_2 and owner iuc.    The tool_dependencies.xml file contains an invalid <repository> tag.Unable to locate repository with 
name package_weblogo_2_8_2 and owner iuc.    The tool_dependencies.xml file contains an invalid <repository> tag.Unable to locate repository with name package_weblogo_2_8_2 and ow
ner iuc.    The tool_dependencies.xml file contains an invalid <repository> tag.Unable to locate repository with name package_weblogo_2_8_2 and owner iuc.    The tool_dependencies
.xml file contains an invalid <repository> tag.Unable to locate repository with name package_weblogo_2_8_2 and owner iuc.    The tool_dependencies.xml file contains an invalid <re
# Snipped a huge error from weblogo

Could not update package_protk_1_2_4
Unable to locate repository with name package_ruby_2_0 and owner bgruening.    The tool_dependencies.xml file contains an invalid <repository> tag.Unable to locate repository with name package_ruby_2_0 and owner bgruening.    The tool_dependencies.xml file contains an invalid <repository> tag.Unable to locate repository with name package_ruby_2_0 and owner bgruening.    The tool_dependencies.xml file contains an invalid <repository> tag.Unable to locate repository with name package_ruby_2_0 and owner bgruening.    The tool_dependencies.xml file contains an invalid <repository> tag.Unable to locate repository with name package_ruby_2_0 and owner bgruening.    The tool_dependencies.xml file contains an invalid <repository> tag.Unable to locate repository with name package_ruby_2_0 and owner bgruening.    The tool_dependencies.xml file contains an invalid <repository> tag.

cd '/home/users/cpt/cpt/esr/Projects/galaxy/tools-iuc/packages/package_pear_0_9_6' && git rev-parse HEAD
cd '/home/users/cpt/cpt/esr/Projects/galaxy/tools-iuc/packages/package_pear_0_9_6' && git diff --quiet
description required for automatic creation of repositories

cd '/home/users/cpt/cpt/esr/Projects/galaxy/tools-iuc/packages/package_ruby2_bioruby_1_4' && git rev-parse HEAD
cd '/home/users/cpt/cpt/esr/Projects/galaxy/tools-iuc/packages/package_ruby2_bioruby_1_4' && git diff --quiet
Could not update package_ruby2_bioruby_1_4
Unable to locate repository with name package_ruby_2_0 and owner bgruening.    The tool_dependencies.xml file contains an invalid <repository> tag.Unable to locate repository with name package_ruby_2_0 and owner bgruening.    The tool_dependencies.xml file contains an invalid <repository> tag.Unable to locate repository with name package_ruby_2_0 and owner bgruening.    The tool_dependencies.xml file contains an invalid <repository> tag.Unable to locate repository with name package_ruby_2_0 and owner bgruening.    The tool_dependencies.xml file contains an invalid <repository> tag.Unable to locate repository with name package_ruby_2_0 and owner bgruening.    The tool_dependencies.xml file contains an invalid <repository> tag.Unable to locate repository with name package_ruby_2_0 and owner bgruening.    The tool_dependencies.xml file contains an invalid <repository> tag.

[bgruening]

That's one reason it's not good to push everything ;)
I will try to fix those ...

[hexylena]

Much appreciated! Feel free to push them whenever you're done with updates

[nsoranzo]

I think that before enabling the automatic push of repos to the MTS we need a way to specify that a directory is in a WIP state, so that the repo is not updated by the bot.

A neat way to do this may be adding a new field to .shed.yml specifying the list of toolsheds to which the repository should be pushed.

[hexylena]

@nsoranzo my way of specifying something is WIP is just by not having it in the master branch. If it's in master, it's assumed to be ready to be published.

[nsoranzo]

@erasche Sorry, bad wording on my part! I mean that if you deem a repository in a good shape for automatic push to the TTS but not to the MTS, there is no way to specify this (unless we create a specific branch for TTS-only pushes).

A nice feature of having a list of toolsheds in .shed.yml is that it would be possible (for other git repositories) to add domain-specific toolsheds or even internal toolsheds for automatic push.

[hexylena]

@nsoranzo hmmm, would you support something like the following then?

• master branch = OK for MTS, TTS
• dev branch = OK for TTS
• other branches/PRs should be merged into dev

I don't disagree that having a feature like that would be nice though :)

[bgruening]

@erasche I think we need some mechanism to blacklist repositories from being pushed to the TS. I have a few example repositories to demonstrate how to use setup_ruby_environment and friends. These and others that are not actively used should not be part of the TS, imho.

[hexylena]

@bgruening example folders...so you mean things that could be gists or embedded into the documentation?

I agree that they shouldn't be part of the TS

[bgruening]

See here for example: https://github.com/galaxyproject/tools-iuc/tree/master/test_repositories

[hexylena]

@bgruening but that's all documentation! I don't believe that that belongs in this repository, or any.

[jmchilton]

I'll implement a conditional switch for for testing/dev repositories in planemo very soon.

[hexylena]

@jmchilton you might implement it per @nsoranzo's suggestion, with the ability to specify TS URLs. It'd be nice to be able to re-use the build/deploy scripts locally for pushing to our TS.

[hexylena]

I'm not against such an option, I just think this is something that has an existing analogue in master/dev branches, and we can leverage those and existing coder conventions to deliniate which tools should be in the MTS, and which in the TTS.

State	Git	Notes	Location
Testing	dev	dev	TTS
Done	master	"production"	MTS, TTS

Merging from dev to master represents promoting things from dev to prod, in DevOps terminology. Promoting individual folders doesn't look to be too complicated

[bgruening]

@erasche imho the IUC repository is to a large degree documentation. Moreover, these repositories had and probably still have the purpose of testing installation recipes.

[hexylena]

@bgruening IMHO that needs to be corrected. Documentation belongs in galaxy's docs folder, the wiki, or the standards repo. If they're test cases for installation recipes, then they should be in a test folder+test case of Galaxy.

However, the majority opinion is yours'/Nicola's, that it would be easiest/best to list TSs that it's OK to push to; I'll be fine with that, I just have very strong opinions about DevOps best practices.

[nsoranzo]

Related questions: what about commits fixing whitespaces or the order of XML elements? Do we want to automatically push also these?
Creating many new repository revisions on the MTS obviously increases the workload on Galaxy admins.

[hexylena]

Good question! I'm in favour of pushing those.

• now that Travis and Jenkins test everything, there shouldn't be too many more times when un-linted code makes it into master
• we could keep better changelogs in readmes, which would help admins determine whether or not to update
• galaxy should eventually auto-update, which would make this point moot.

[peterjc]

@erasche's arguments on having a mapping from branch names to tool sheds is interesting - but I think it should be configurable (e.g. I might want to use master and stable, or master and release).

On the other hand, I strongly favour keeping documentation for tools in a repository in that same repository, otherwise it is all to easy to let it fall out of sync.

[hexylena]

@peterjc I've conceded the point, we'll just use something like galaxyproject/planemo#166 and that'll make people happy.

I'm likewise strongly in favour of keeping documentation in repo, but in the correct repo. I still believe that everything in the test_repositories folder is galaxyproject/galaxy docs folder (or wiki) type documentation and not something that belongs in a tool git repo. I'd be happy to be convinced otherwise.

[hexylena]

@peterjc - in that unused proposition, the configurability for branches would come in in your CI server, just set it up to run TTS upload/MTS upload on whichever branches you want. It wouldn't be part of planemo/the repo itself.

[peterjc]

In an experimental branch https://github.com/peterjc/galaxy_mira/tree/planemo_ci of https://github.com/peterjc/galaxy_mira I have been trying the following:

• planemo lint
• planemo diff (vs testtoolshed)
• planemo test (only those with changes)
• planemo shed_update -t testtoolshed (only those with changes)

This is in part to cut the continuous integration test run-time which was becoming a problem on my larger tool collection repositories (and would prevent running all the tool tests on https://github.com/galaxyproject/tools-iuc or https://github.com/galaxyproject/tools-devteam too), but the final step of automatically pushing to the TTS matches this issue.

[mvdbeek]

@peterjc

This is similar to how I am handling the things for https://github.com/ARTbio/tools-artbio, except that I'm doing a shed_test instead of test (and I broke it recently ;)). Around GCC time i already did this for devteam and tools-iuc, and about 20% of tests were failing due to erroneous tool_dependencies.xml .
Before we do automatic uploads, we should address this as well.