ENH move to glibc 2.17 (aka cos7)

[xhochy]

To do:

• split out the sysroot in the current ct-ng packages and use CDT sysroot (ENH split the sysroot ctng-compilers-feedstock#23)
• remove cos from the sysroot name for binutils (ENH changes for split sysroot binutils-feedstock#25)
• make new binutils not like old compilers (Prevent installing with older gcc builds binutils-feedstock#27)
• patch the old binutils packages to have them be incompatible with the sysroot packages (ENH make old binutils and associated packages conflict with new sysroot conda-forge-repodata-patches-feedstock#49)
• fix issue with track features and current_repodata.json (ENH make sure packages in current repo data w/ features have versions without features conda/conda-build#3957, ENH v2.12 priority linux-sysroot-feedstock#12, ENH fix 2.17 sysroots linux-sysroot-feedstock#11)
• build the cos7 glibc 2.17 CDT
• test compiler w/ new cos7 CDT
• make sure the proper virtual package constraints are present in the CDT package and exported to the proper runtimes
• fix extra DSO errors (ENH check all sysroot locations for DSOs conda/conda-build#3969)
• merge final PRs for sysroot shims (ENH fix the sysroot shims binutils-feedstock#32, [DO NOT MERGE] WIP try the new variants ctng-compilers-feedstock#29)
• mark the latest version of the compilers as broken to move to new sysroot (ENH mark compilers broken for new sysroot admin-requests#67)
• ship 9.3 libs for ctng-compilers feedstock based on new sysroot build (ENH build 9.3.0 ctng-compilers-feedstock#34)
• fix issue with virtual package constraints (cudatoolkit run constrain on __cuda not used for solve conda/conda#9896)
• Bring all x86_64 cos6 CDTs into conda-forge
• Bring all x86_64 cos7 CDTs into conda-forge
• After the 2 above, move CDTs into conda symlink
• make old compiler activation scripts conflict with new compilers (ENH make ctng activation conflict with new compilers conda-forge-repodata-patches-feedstock#68)
• add sysroot deps to new compiler activation scripts (ENH use new sysroot ctng-compiler-activation-feedstock#24)
• make sure to add a sysroot run dep to gcc_impl, gxx_impl and gfortran_impl so that the repo data patches for the CDTs don't break the new compilers (ENH add extra sysroot deps to mark builds ctng-compilers-feedstock#32)
• patch conda-build for sysroot distro name (ENH enable custom build distros conda/conda-build#3999)
• fix python sysconfig data

stretch goals

• maybe build gcc from a new gcc-feedstock instead of ct-ng?

other relevant issues

• uninstalling from symlinks in other packages is broken (noarch: python packages incorrectly handled when using pypy conda/conda#9810 (comment))

I (@beckermr) am pretty out of my depth here but ready to learn!

old comment from @xhochy

I have packages that require newer glibc features like setns or process_vm_readv. These are not included in the glibc version currently used for building on Linux. Is there a way to use a newer baseline glibc* for a package or impossible until we up the base CentOS version globally?

*I'm aware that the package would run on less systems but that would do it either way.

[isuruf]

It's not possible at the moment. There are several steps involved,

1. Add a virtual package to conda. This is discussed in Packages that require new OSX SDKs #891 for newer OSX SDKs as well.

2. Add a run_exports type for run_constrained. (This is optional) This is so that the virtual package is added as a run_constrained automatically.

3. Build new compilers.

4. Add a new docker image.

[jakirkham]

Raised issue ( conda/conda#9348 ) to discuss 1.

[jakirkham]

We had a discussion similar to 2 for cudatoolkit. ( conda/conda#9115 )

[mingwandroid]

Pinging @kalefrannz and @msarahan for comments too. We had specced out some metadata for the solver, is @isuruf's plan enough here? Do we need virtual packages?

[msarahan]

Virtual packages are probably not strictly necessary, but they will make it much easier to tell people that have inadequate glibc exactly why they can't install a package. It probably needs to be two virtual packages, though: one for libc implementation (glibc, musl) and another for version.

[isuruf]

It probably needs to be two virtual packages, though: one for libc implementation (glibc, musl) and another for version.

Can't it be one each for libc implementation? A virtual package like glibc=2.19 would do right? We don't need mutual exclusivity because the system ensures that.

[jakirkham]

We don't need mutual exclusivity because the system ensures that.

Not exactly. glibc and musl do have some level of compatibility. However this isn't always the case and when there is an issue it can be quite subtle. Would much rather have two packages like Mike suggests.

[scopatz]

@isuruf - what needs to be done for (3) build newer compilers at this point?

[isuruf]

The sysroot for the compiler is built at https://github.com/conda-forge/ctng-compilers-feedstock
We need to change the sysroot from 2.12.2 to 2.17. To ensure that we don't use 2.17 by default, we first have to split the sysroot to another output, call it sysroot_cos6_linux-64 and then pin that in the activation scripts. After that we can bump the sysroot to 2.17.

[scopatz]

@isuruf - how are we suppossed to select between the different sysroots? It seems that both of them have the [linux64] selector. Maybe another question is do we intend to build both the centos6 & centos7 stacks simeltaneously as different ouptuts, or is the intention that we simply flip the sysroot version number whenever we want to build a different version?

[isuruf]

@scopatz, I'm sorry I don't understand. Can you explain more?

[scopatz]

I don't fully understand what to do. I guess I can try putting in a PR and you can give feedback / push to that branch? 😉

[isuruf]

@conda-forge/core, the compilers unfortunately have conda_cos6 name in them and some packages hardcode this name for later use.

There are two options,

1. Rename to conda_cos and have a symlink from conda_cos6 to conda_cos
2. Have conda_cos7 and have a symlink from conda_cos6 to conda_cos7

1 makes it easier.
2 is also fine and has the added benefit that packages which were built with conda_cos7 will require a new sysroot instead of possibly silently using the conda_cos6 sysroot, but when conda_cos8 comes along we need 2 sets of symlinks.

Which option should we go with? Since this affects defaults, pinging @kalefranz, @mingwandroid

[scopatz]

I think we should go with the most future-proof way of handling this, which seems to be option 2.

[isuruf]

@kalefranz, @mingwandroid, @msarahan, @jjhelmus, @nehaljwani, need your feedback too as we want this to be consistent with defaults.

[msarahan]

Option 2 seems the wisest.

[nehaljwani]

My personal opinion is to drop the distro name from the sysroot completely. I don't see any reason for associating a substring of HOST/BUILD with 'CentOS'. The pseudo cross compiler targets a particular kernel version and glibc version, not a Linux distribution.

One of the reasons why I say that is because CentOS folks maintain a huge number of patches over glibc, which crosstool-ng doesn't. For example, they back-ported IP_MULTICAST_ALL to glibc 2.12, which crosstool-ng didn't. So using cos6 was also misleading IMO. Also, if, in the future, CentOS decides to become more bleeding edge, and is no longer the widely used distro with the oldest glibc, then we might want to switch to debian/opensuse's glibc as the baseline version, then it would become even harder to get rid of 'cosX' in the name.

[mingwandroid]

For example, they back-ported IP_MULTICAST_ALL to glibc 2.12, which crosstool-ng didn't.

The first version of these compilers targeted CentOS5. I took care to port all patches at that time. I consider this a bug and we should be aim to port the CentOS6 patches across.

An alternative, which might be faster would be to use CDT packages for glibc, glibc-devel, kernel-headers and what not. That should work.

[scopatz]

It still is not entirely clear what to do here...

[isuruf]

conda-forge/staged-recipes#10479 adds the sysroot from CDT packages. After merging that PR, the compiler package should remove building glibc and depend on that package.

[scopatz]

I see that makes sense

[jjhelmus]

comment moved to staged-recipes PR.

[jack-pappas]

The staged-recipes PR was merged back on 2020-01-20. Does that mean conda-forge/ctng-compilers-feedstock#3 can move forward using the last suggestion from @isuruf ?

[jakirkham]

Just a note, we ran into a bit of surprise recently with the __cuda virtual package when trying to constrain cudatoolkit with it ( conda/conda#9115 (comment) ). Not sure if this extends to __glibc or __osx when constraining compiler runtimes with them. Figured it was worth mentioning just in case.

[beckermr]

I am getting interest from ppl I work with on seeing the move to 2.17 go through. Also, I think rhel6 is sunsetting in November. Our CDTs could not receive updates after that in principle at least. We may not actually update the CDTs OFC.

[isuruf]

WIP stuff at
https://github.com/isuruf/binutils-feedstock/tree/name
https://github.com/isuruf/gcc-feedstock

Feel free to take over

[beckermr]

Yup that issue @jakirkham. Very glad that you are willing to look into it!

[beckermr]

Ok everyone. The compilers with cos7 are live. The docs pr is up to.

If anyone here wants to try them out, that’d be great!

[leofang]

@beckermr Great! Is it possible now to build packages that only support glibc 2.17? If so, how and where do I add the constraint __glibc==2.17?

[beckermr]

See the announcement on the docs page. You'll pick up a slightly older libcxx and libgcc but this should be fixed in a day or so

[xhochy]

@beckermr Using this in the feedstock for which this issue was originally written: conda-forge/py-spy-feedstock#6

conda smithy rerender is failing for me locally with:

% conda smithy rerender
No azure token. Create a token and
put it in ~/.conda-smithy/azure.token
INFO:conda_smithy.configure_feedstock:Downloading conda-forge-pinning-2020.07.26.20.00.32
INFO:conda_smithy.configure_feedstock:Extracting conda-forge-pinning to /var/folders/3r/k0d_q3hs2mb2_jxfh_9hjyqr0000gn/T/tmprnlbfuv0
Setting build platform. This is only useful when pretending to be on another platform, such as for rendering necessary dependencies on a non-native platform. I trust that you know what you're doing.
WARNING:conda_build.config:Setting build platform. This is only useful when pretending to be on another platform, such as for rendering necessary dependencies on a non-native platform. I trust that you know what you're doing.
Setting build arch. This is only useful when pretending to be on another arch, such as for rendering necessary dependencies on a non-native arch. I trust that you know what you're doing.
WARNING:conda_build.config:Setting build arch. This is only useful when pretending to be on another arch, such as for rendering necessary dependencies on a non-native arch. I trust that you know what you're doing.
No numpy version specified in conda_build_config.yaml.  Falling back to default numpy value of 1.11
WARNING:conda_build.metadata:No numpy version specified in conda_build_config.yaml.  Falling back to default numpy value of 1.11
Adding in variants from internal_defaults
INFO:conda_build.variants:Adding in variants from internal_defaults
Adding in variants from /var/folders/3r/k0d_q3hs2mb2_jxfh_9hjyqr0000gn/T/tmprnlbfuv0/conda_build_config.yaml
INFO:conda_build.variants:Adding in variants from /var/folders/3r/k0d_q3hs2mb2_jxfh_9hjyqr0000gn/T/tmprnlbfuv0/conda_build_config.yaml
Adding in variants from /Users/uwe/Development/py-spy-feedstock/recipe/conda_build_config.yaml
INFO:conda_build.variants:Adding in variants from /Users/uwe/Development/py-spy-feedstock/recipe/conda_build_config.yaml
Traceback (most recent call last):
  File "/Users/uwe/miniconda3/bin/conda-smithy", line 10, in <module>
    sys.exit(main())
  File "/Users/uwe/miniconda3/lib/python3.7/site-packages/conda_smithy/cli.py", line 592, in main
    args.subcommand_func(args)
  File "/Users/uwe/miniconda3/lib/python3.7/site-packages/conda_smithy/cli.py", line 398, in __call__
    self._call(args, tmpdir)
  File "/Users/uwe/miniconda3/lib/python3.7/site-packages/conda_smithy/cli.py", line 409, in _call
    temporary_directory=temporary_directory,
  File "/Users/uwe/miniconda3/lib/python3.7/site-packages/conda_smithy/configure_feedstock.py", line 1898, in main
    render_azure(env, config, forge_dir, return_metadata=True)
  File "/Users/uwe/miniconda3/lib/python3.7/site-packages/conda_smithy/configure_feedstock.py", line 1190, in render_azure
    return_metadata=return_metadata,
  File "/Users/uwe/miniconda3/lib/python3.7/site-packages/conda_smithy/configure_feedstock.py", line 570, in _render_ci_provider
    os.path.join(forge_dir, forge_config["recipe_dir"]), config=config
  File "/Users/uwe/miniconda3/lib/python3.7/site-packages/conda_build/variants.py", line 525, in get_package_combined_spec
    combined_spec = combine_specs(specs, log_output=config.verbose)
  File "/Users/uwe/miniconda3/lib/python3.7/site-packages/conda_build/variants.py", line 277, in combine_specs
    log_output=log_output)
  File "/Users/uwe/miniconda3/lib/python3.7/site-packages/conda_build/variants.py", line 252, in _combine_spec_dictionaries
    spec_source))
ValueError: variant config in /Users/uwe/Development/py-spy-feedstock/recipe/conda_build_config.yaml is ambiguous because it does not fully implement all zipped keys, or specifies a subspace that is not fully implemented

[mingwandroid]

@kkraus14, is there any way to get any insight into why and when the base glibc version for CUDA gets updated? It's something that creates a lot of uncertainty as things stand.

[beckermr]

@xhochy, see the PR. We have zipped docker image with cuda_compiler_version so we need this

cuda_compiler_version:                    # [linux64]
  - None                                  # [linux64]
docker_image:                             # [linux64]
  - condaforge/linux-anvil-cos7-x86_64    # [linux64]

[xhochy]

@xhochy, see the PR. We have zipped docker image with cuda_compiler_version so we need this

cuda_compiler_version:                    # [linux64]
  - None                                  # [linux64]
docker_image:                             # [linux64]
  - condaforge/linux-anvil-cos7-x86_64    # [linux64]

This needs to be added to https://conda-forge.org/docs/maintainer/knowledge_base.html#centos7 then

[beckermr]

Yup! I was about to do that. :)

[beckermr]

#1118

[kkraus14]

@kkraus14, is there any way to get any insight into why and when the base glibc version for CUDA gets updated? It's something that creates a lot of uncertainty as things stand.

It's typically tied to the supported CentOS / RHEL version and I believe that roughly aligns to the EOL for that distribution, but there's nothing formal AFAIK.

[beckermr]

I found a problem where conda is setting the build machine using the old sysroots. Patch submitted here: conda/conda-build#3999

We'll need to patch the CI setup scripts to set this env var as well.

[jakirkham]

Is there a good way to specify the CDT's include and lib directories? Perhaps an environment variable?

[beckermr]

Is there a good way to specify the CDT's include and lib directories? Perhaps an environment variable?

What? They are in the sysroot so they should just work I thought...

[beckermr]

Error logs?

[jakirkham]

The build in particular looks in /usr by default. So we have to override the path to look at the right path specifically. Just asking to figure out how we do that.

[isuruf]

Please open a different issue. This is about supporting glibc 2.17

[jakirkham]

Right I'm trying to do this as part of a CentOS 7 build. There's no issue on CentOS 6.

[isuruf]

Please open another issue with conda list information and details on how to reproduce.

[jakirkham]

Just asking whether there is an environment variable (maybe CONDA_BUILD_SYSROOT?) or similar that points to the location of the CDTs once installed during a build. Am hoping this is an easy answer that someone here knows 🙂

[isuruf]

Yes CONDA_BUILD_SYSROOT is there, but it shouldn't be needed.

[mingwandroid]

CONDA_BUILD_SYSROOT is macOS only. I'm not opposed to setting it on Linux, but as @isuruf says its not needed. The compilers hard-code the relative path (kind-of) to the sysroot, and the sysroots are searched. If you pass e.g. -I/usr/include or -L/usr/lib then you are doing it wrong, those are not needed and prevent cross-compilation from working correctly.

[jakirkham]

Great, thanks! 😄

I'm not passing those flags, but the build's configure scripts search in /usr by default and needs to be overridden. So we are trying to wrangle it into doing the right thing. Agree it would be nice if we didn't have to (would make my life easier 😅)

[mingwandroid]

Build configure scripts should not add system paths, the compilers already know those. Good luck fixing this!

[beckermr]

OK. cos7 is done and fully shipped. I know of no more PRs that need to be made. A few need to be merged, but they are not essential. Closing!