Add to conda-forge

[matthewfeickert]

It would be nice to also have IRIS-HEP tools available on conda-forge.

For servicex there's some additional work in other areas though that would be required as using grayskull to check we find

$ grayskull pypi --list-missing-deps --strict-conda-forge 'servicex==3.0.0a12'



#### Initializing recipe for servicex==3.0.0a12 (pypi) ####

Recovering metadata from pypi...
Starting the download of the sdist package servicex
servicex 100% Time:  0:00:00   1.7 MiB/s|#################################################################################################################################################################|
Checking for pyproject.toml
pyproject.toml found in /tmp/grayskull-servicex-u73ng_33/servicex-3.0.0a12/pyproject.toml
Recovering information from setup.py
Executing injected distutils...
Checking >> types-pyyaml 100% |####################################################################################################################################################|[Elapsed Time: 0:00:06]
Recovering license info from spdx.org ...
Matching license file with database from Grayskull...
Match percentage of the license is 97%. Low match percentage could mean that the license was modified.
License type: BSD-3-Clause
License file: ['LICENSE']
Build requirements:
  <none>
Host requirements:
  - python >=3.8
  - hatchling >=1.13.0
  - pip
Run requirements:
  - python >=3.8
  - qastle >=0.17
  - func-adl >=3.2.6
  - requests >=2.31
  - pydantic >=1.10,<2.0
  - httpx >=0.24
  - miniopy-async >=1.15
  - tinydb >=4.7
  - google-auth >=2.17
  - typer >=0.9.0
  - pyyaml >=6.0
  - types-pyyaml >=6.0
  - rich >=13.0.0
  - nest-asyncio >=1.5.7
  - aiofile
  - make-it-sync

RED: Missing packages
GREEN: Packages available on conda-forge
Missing dependencies: qastle, make-it-sync, func-adl, miniopy-async

Maintainers:
   - matthewfeickert

#### Recipe generated on /tmp for servicex==3.0.0a12 ####

that qastle, make-it-sync, func-adl, miniopy-async are all missing from conda-forge.

I can follow up with @gordonwatts and @masonproffitt on if they're okay with me packaging func-adl, qastle, and make-it-sync for conda-forge. miniopy-async isn't missing any packages, so we could ask if they would consider distributing on conda-forge too. (Done in hlf20010508/miniopy-async#25)

[matthewfeickert]

conda-forge/staged-recipes#25848 added https://github.com/conda-forge/miniopy-async-feedstock so miniopy-async is now packaged on conda-forge.

[matthewfeickert]

Now just missing func-adl and qastle

$ grayskull pypi --list-missing-deps --strict-conda-forge 'servicex==3.0.0a15'



#### Initializing recipe for servicex==3.0.0a15 (pypi) ####

Recovering metadata from pypi...
Starting the download of the sdist package servicex
servicex 100% Time:  0:00:00   4.9 MiB/s|#################################################################################################################################################################|
Checking for pyproject.toml
pyproject.toml found in /tmp/grayskull-servicex-w83ffbke/servicex-3.0.0a15/pyproject.toml
Recovering information from setup.py
Executing injected distutils...
Checking >> types-pyyaml 100% |####################################################################################################################################################|[Elapsed Time: 0:00:06]
Recovering license info from spdx.org ...
Matching license file with database from Grayskull...
Match percentage of the license is 97%. Low match percentage could mean that the license was modified.
License type: BSD-3-Clause
License file: ['LICENSE']
Build requirements:
  <none>
Host requirements:
  - python >=3.8
  - hatchling >=1.13.0
  - pip
Run requirements:
  - python >=3.8
  - qastle >=0.17
  - func-adl >=3.2.6
  - requests >=2.31
  - pydantic >=1.10,<2.0
  - httpx >=0.24
  - miniopy-async >=1.15
  - tinydb >=4.7
  - google-auth >=2.17
  - typer >=0.9.0
  - pyyaml >=6.0
  - types-pyyaml >=6.0
  - rich >=13.0.0
  - aiofile
  - make-it-sync

RED: Missing packages
GREEN: Packages available on conda-forge
Missing dependencies: func-adl, qastle

Maintainers:
   - matthewfeickert

#### Recipe generated on /tmp for servicex==3.0.0a15 ####


$ 

with PRs opened to help address this:

• feat: Add LICENSE iris-hep/func_adl#144
• feat: Add qastle conda-forge/staged-recipes#26178

[gordonwatts]

Quick question about this, @matthewfeickert - these are pure python packages - what is the advantage of also maintaining a package there, vs. just having them in pypi and pulling them from there (which I think conda's various package managers can now)? I'm asking because the more moving parts there are, the more likely it is things will get out of sync.

[matthewfeickert]

@gordonwatts Good questions.

these are pure python packages - what is the advantage of also maintaining a package there, vs. just having them in pypi and pulling them from there (which I think conda's various package managers can now)?

Conda requires all dependency packages to already exist somewhere as conda-packages for you to be able to package something as a conda package. So to be able to package the serviceX client you need all its dependencies on conda-forge.

As to "why package the serviceX client on conda-forge at all if it is pure-Python?", the only conda package manager that truly resolves things with equal parity between conda package indexes (e.g. conda-forge) and pip package indexes (e.g. PyPI) is pixi, so in general if you have people using a conda family tool, you want them to be able to interact with conda-forge as much as possible and only fall back to PyPI when needed.

It is true that in most situations in which you're dealing with things on PyPI you're not going to run into problems, but providing these simple repackagings also cuts down on (or fully eliminates) the package resolution that your package manager of choice needs to do between the different metadata trees.

So this is not necessary, but it does makes things easier, especially the more complex an environment gets where incorrect solves can start to happen.

I'm asking because the more moving parts there are, the more likely it is things will get out of sync.

Not really (though I mean, yes, that is just a true statement about things). Even if the human in the loop reviewing the autogenerated PRs is considered the problem/blocker here you can enable automerge so that conda-forge just keeps your tools updated as long as they build. It is perhaps better to think about having a package feedstock on conda-forge being controls that you hook into the conda-forge framework than what we do with distributing packages to PyPI where you have to come with your own packaging system and then just distribute the resulting artifacts.

I will give you though that if you have a complex system like scikit-hep/fastjet (https://github.com/conda-forge/fastjet-feedstock) that is forced to use older build systems then adding initial support to conda-forge can be a huge pain (conda-forge/staged-recipes#21052). Though, as you noted, for these pure-Python packages it is basically a single command line call and you're good to go.

[matthewfeickert]

conda-forge/staged-recipes#26178 has been merged and so there is now https://github.com/conda-forge/qastle-feedstock. The only remaining thing is for @gordonwatts to make a new func_adl release with the license packaged (iris-hep/func_adl#144 (comment)).

[matthewfeickert]

Now everything is on conda-forge, but ruamel_yaml is abandoned and so way behind the required versions (c.f. conda-forge/ruamel_yaml-feedstock#107).

edit: Ah want to use https://github.com/conda-forge/ruamel.yaml-feedstock/ instead.

[matthewfeickert]

Adding with conda-forge/staged-recipes#27997