AVX support

[petrkozorezov]

By default, polars python library builds with AVX/AVX2/... support and there is a separate version for legacy processors.
I checked explorer code and didn't found any mentions of AVX. So a tried to build explorer with RUSTFLAGS from polars and saw 4% performance increasing in my own code.
So maybe follow polars python library way: prebuild two versions of explorer (with (by default) and without AVX support) and switch between them with a flag (or smth like that)?

How I built explorer with AVX support:

  Mix.install(
    [
      {:rustler, ">= 0.0.0", optional: true},
      {:explorer, "0.5.4"},
    ],
    system_env: [
      EXPLORER_BUILD: "1",
      RUSTFLAGS: "-C target-feature=+fxsr,+sse,+sse2,+sse3,+ssse3,+sse4.1,+sse4.2,+popcnt,+avx,+fma",
    ]
  )

UPD: Enabling AVX2 adds me another 1% of performance.

[josevalim]

Sounds good to me! We may need some changes in Rustler precompiled so we include this as a flag too.

[philss]

Hey @petrkozorezov , there is a PR adding this feature: #710

Would you mind to share your opinions? Do you think it covers what you proposed?

I couldn't cover all the targets that run in x86_64 because some of them didn't compile with the flag (Apple one), or I couldn't easily detect if the target was running with the features needed enabled. But for the later, a compile env (or a env var) can be used to turn on the "legacy mode". WDYT?

[petrkozorezov]

Hey @petrkozorezov , there is a PR adding this feature: #710

Would you mind to share your opinions? Do you think it covers what you proposed?

I couldn't cover all the targets that run in x86_64 because some of them didn't compile with the flag (Apple one), or I couldn't easily detect if the target was running with the features needed enabled. But for the later, a compile env (or a env var) can be used to turn on the "legacy mode". WDYT?

Looks great! But it's a little bit unclear (without going deeper into the code) how to use legacy mode build (and that it's even exists). Maybe add a few words about it to README?

[philss]

@petrkozorezov Thanks! I added a note to the README.md.

I'm going to hold the PR until we decide to launch a new version, since this is going to be a breaking change due to the dependency on RustlerPrecompiled ~> 0.7.

[josevalim]

@philss we can merge if you want. tokenizers depends on ~> 0.6, so it should work with ~> 0.7 as well just fine. :)

[philss]

@josevalim done! Thanks!