The Conda package doesn't work on CentOS 7 and Ubuntu 18.04

[bryant1410]

import tokenizers doesn't work on CentOS 7 (and RHEL 7), because it has glibc 2.17, while the Conda package (at least for Python 3.8) was compiled against a newer version (the error says version 'GLIBC_2.18' not found).

I believe CentOS and RHEL 7 are both still popular, though I can't back up this claim (I can't find a link). My university cluster uses it at least. And note version 7 is the latest CentOS version with long-term support.

It does work from PyPI's version, so I just use that one. However, it'd be good it worked from Conda. So it seems it's related to the glibc version specified by Conda, while the one provided by ubuntu-latest (Ubuntu 18.04 as of today, IIUC) seems to work fine. Conda docs say they do provide their own glibc version. Because there isn't such a libc package in Conda, they provide a solution via virtual packages. So it seems that all that's needed is to set an env var like:

CONDA_OVERRIDE_GLIBC=2.17

(note Rust dynamically links the compiled binary to the available glibc version at build time).

Could you take a look at it? Not sure how it can be tested w/o pushing a release.

Related issues (some people also mention CentOS 7):

• Minimum Supported glibc version rust-lang/libc#1412
• How to compile rust with a specific GLIBC version for gnueabihf architecture?
• Rust binaries require a too recent glibc rust-lang/rust#57497

Update: it's got a bit worse, as now it requires a newer GLIBC version:

/lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.29' not found

It's not working for me on Ubuntu 18.04 (GLIBC 2.27).

What this article details may be the solution for it: Building Rust binaries in CI that work with older GLIBC

[freddycct]

I face the same issue.

[Rictus]

I am facing the same issue.

[harrisonbay]

Yep, same issue here as well on CentOS 7.9 (used in my university's cluster)

[AmenRa]

Same issue on Ubuntu 18.04.5 LTS.
Ubuntu 18.04's latest GLIBC version is 2.27.
Conda-installed tokenizers (through transformers installation) version 0.10.2 requires GLIBC 2.29.

[rubencart]

Same thing as @AmenRa

[e-yi]

Same issue. Had to uninstall tokenizers temporarily.

[AmenRa]

Same issue on Ubuntu 18.04.5 LTS.
Ubuntu 18.04's latest GLIBC version is 2.27.
Conda-installed tokenizers (through transformers installation) version 0.10.2 requires GLIBC 2.29.

As a workaround, I've installed the previous tokenizers version, and everything works fine now:

conda install -c huggingface tokenizers=0.10.1 transformers=4.4.2

[zack-kimble]

Same issue for me on Ubuntu 18.04. I just changed to pip installation, which seems to be working.

[Leojc]

use pip instead of conda:

conda uninstall tokenizers, transformers
pip install transformers

[bryant1410]

This article explains how to fix it, if anybody wants to go ahead: https://kobzol.github.io/rust/ci/2021/05/07/building-rust-binaries-in-ci-that-work-with-older-glibc.html

[brando90]

same issue here: huggingface/transformers#15062

[Narsil]

Hi @brando90 do you mind creating a new issue with exactly what you are trying and what's failing and what you expect ?

This issue and the linked one seem like two very different kind of issues.

Cheers.

[brando90]

Sure! #1080 <#1080>

…

----- Brando Miranda

On Oct 5, 2022, at 6:29 AM, Nicolas Patry ***@***.***> wrote: Hi @brando90 <https://github.com/brando90> do you mind creating a new issue with exactly what you are trying and what's failing and what you expect ? This issue and the linked one seem like two very different kind of issues. Cheers. — Reply to this email directly, view it on GitHub <#585 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAOE6LXPURNA7R3TZNRM5P3WBV7DNANCNFSM4VZOSHJQ>. You are receiving this because you were mentioned.

[Narsil]

Closing this

[bryant1410]

@Narsil is this really fixed? When? How? Can't test it right now.