python38Packages.tokenizers fails to build, possibly since the last rustc update?

[mstone]

Describe the bug
python38Packages.tokenizers, which is depended on by python38Packages.transformers, has been failing to build on both Linux and macOS since build 123843520 / scmdiff, perhaps since rustc was updated in 1f9cd4c.

Logs of a contemporary build attempt indicate that the immediate problem is that the current version of tokenizers depends on a vendored copy of v0.10.0 of the parking_lot crate, via its cargoDeps dependency.

Further research suggests that the asm! -> llvm_asm! issue reported in the logs was fixed in Amanieu/parking_lot#223 which it looks like is included in v0.10.1 and successors.

As a result, I think that someone just needs to refresh tokenizers' cargoDeps (possibly by way of simply updating the package) or, failing that, to try to get it to build with an older version of rustPlatform.

(That said, I have yet to test these changes myself, so I don't know if they are truly enough or if additional modifications are required.)

To Reproduce

nix -L build nixpkgs#python38Packages.tokenizers

Expected behavior
The build should succeed.

Notify maintainers
@danieldk

(also /cc @alyssais (since this looks like an interaction with rustc and since there is a commit linked to above updating rustc in the scmdiff from the last successful build to the first failing build, per hydra.)

Metadata
$ nix-shell -p nix-info --run "nix-info -m"

• system: "x86_64-darwin"
• host os: Darwin 19.5.0, macOS 10.15.5
• multi-user?: no
• sandbox: no
• version: nix-env (Nix) 2.4pre20200714_832e111
• channels(...): "nixpkgs-20.09pre221814.10100a97c89"
• nixpkgs: /Users/.../.nix-defexpr/channels/nixpkgs

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute: python38Packages.tokenizers

[danieldk]

Thanks for the bug report (still hoping for Hydra notifications some day)! The linked PR should fix this issue.

I am also working on some other PRs to make transformers work on Linux/AArch64 and Darwin/x86_64.