Allow collections to install RPMs from other sources (like URLs)

[felixfontein]

I'm trying to add EE support to community.sops (ansible-collections/community.sops#98). For that I want to install Mozilla's sops (https://github.com/mozilla/sops/) in the EE image. Sops RPMs are available from GitHub (https://github.com/mozilla/sops/releases).

If it is already possible to install an RPM from a file I didn't manage to figure out how (my first attempt of specifying the URL in the bindep requirements file failed miserably and triggered #90: https://github.com/ansible-collections/community.sops/runs/6039420137?check_suite_focus=true). But maybe I missed something?

[eqrx]

Hey, thanks for opening the issue! I would suggest adding a custom build step as described in https://ansible-builder.readthedocs.io/en/stable/definition.html#additional-custom-build-steps here. This gives you the flexibility to add whatever however you like.

If you would like to specifically add RPM via bindep I would suggest opening an issue with them since that they do not allow installing arbitrary RPMs.

[felixfontein]

@eqrx I tested that, but that did not work. ansible-builder only uses dependencies from meta/execution-environment.yml in a collection (https://github.com/ansible/ansible-builder/blob/devel/ansible_builder/introspect.py#L144-L149). additional_build_steps is only available in the user's definition of the EE. Forcing everyone who wants to include community.sops into an EE to manually add additional_build_steps is not a good idea.

[Shrews]

Hi @felixfontein. I think I understand your use-case from a collection perspective.

The ideal solution here would be to have bindep understand how to do this. ansible-builder is just the glue to call different installers (pip, ansible-galaxy, bindep) to handle package installation. I'm not certain making builder understand how to do package installation (either rpms, debs, etc) is the best path forward on this. That would add tons of complexity to a rather simple product.

Allowing collections to specify additional_build_steps might work, but then that, too, could get complicated since a collection isn't going to really know what base OS they are using, are they? Not to mention interaction with other steps from other collections.

I can't really think of a good solution here, other than support at the bindep level.

[felixfontein]

I created https://storyboard.openstack.org/#!/story/2010005 for this.

[cboylan]

Note that Bindep is not an installer. All Bindep does is check the current list of packages and tell you what you need to install to match the listed set. How you install the set and where you get those packages from is not part of what Bindep manages. To be clear I don't think this is something that Bindep would do as it doesn't install anything.

[felixfontein]

@cboylan bindep does choke on URLs though:

Traceback (most recent call last):
  File "/usr/local/bin/bindep", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.8/site-packages/bindep/__main__.py", line 56, in main
    depends = bindep.depends.get_depends(
  File "/usr/local/lib/python3.8/site-packages/bindep/depends.py", line 74, in get_depends
    return Depends(fd.read(), filename)
  File "/usr/local/lib/python3.8/site-packages/bindep/depends.py", line 147, in __init__
    self._rules = parser.rules()
  File "/usr/local/lib/python3.8/site-packages/parsley.py", line 98, in invokeRule
    raise err
ometa.runtime.ParseError: 
https://github.com/mozilla/sops/releases/download/v3.7.2/sops-3.7.2-1.x86_64.rpm [platform:rpm]  # from collection community.sops
      ^
Parse error at line 1, column 6: expected EOF. trail: [version comment rule]

So some minimal changes to bindep (namely: accept URLs for package names) are probably necessary.

[cboylan]

No, what you would do is list sops in your bindep file. Then bindep would check with the package manager to see if a package called sops is present or not. If not it would return it as a missing package. Then the process of installing it and where to install it from is external to bindep. It isn't valid to encode that in bindep.

[felixfontein]

That would require ansible-builder to have some functionality where it is possible to specify that the sops package can be installed by doing xxx on Linx distribution Y, with optional version Z, ... So basically ansible-builder would have to handle profile matching as well?

[cboylan]

I'm not familiar with ansible-builder so I'm not sure. But the bindep tool isn't set up to manage package sources or installation methods. Its primary utility is to compare list of packages against what is in the system and give an answer back as to what is missing. Then the process of reconciling that delta is something the bindep user needs to sort out.

We primarily use it to express package dependencies across a number of platforms so that if I'm going to build software on Ubuntu then go to Rocky Linux then Tumbleweed I can easily determine what packages I am missing regardless of platform.

[felixfontein]

I unfortunately also don't know how exactly ansible-builder and bindep interact. @eqrx or @Shrews should have a better understanding of this, maybe they have an idea?

[felixfontein]

I think that the main work is done by the scripts in https://github.com/ansible/python-builder-image/tree/main/scripts. These scripts seem to end up in the ansible-runner images (coming form the python-builder image built from that repository). Right now it looks like these files are tightly coupled to ansible-builder, and I guess any change in this repo that would help for this issue would need corresponding changes there as well.