[2.0] Adopt interleaved_matmul_* into npx namespace

[barry-jin]

Description

Part1 of feature request #20373

Checklist

Essentials

• PR's title starts with a category (e.g. [BUGFIX], [MODEL], [TUTORIAL], [FEATURE], [DOC], etc)
• Changes are complete (i.e. I finished coding on this PR)
• All changes have test coverage
• Code is well-documented

Changes

• Feature1, tests, (and when applicable, API doc)
• Feature2, tests, (and when applicable, API doc)

Comments

• If this change is a backward incompatible change, why must this change be made.
• Interesting edge cases to note here

[mxnet-bot]

Hey @barry-jin , Thanks for submitting the PR
All tests are already queued to run once. If tests fail, you can trigger one or more tests again with the following commands:

• To trigger all jobs: @mxnet-bot run ci [all]
• To trigger specific jobs: @mxnet-bot run ci [job1, job2]

---

CI supported jobs: [website, sanity, edge, clang, windows-gpu, miscellaneous, centos-cpu, unix-cpu, windows-cpu, unix-gpu, centos-gpu]

---

Note:
Only following 3 categories can trigger CI :PR Author, MXNet Committer, Jenkins Admin.
All CI tests must pass before the PR can be merged.

[ptrendx]

Would it be more beneficial to have the full multihead attention primitive exposed in the API instead (it could still be implemented by the interleaved matmuls)? Both Keras and pyTorch have it as a layer, and this could make the usage easier. What do you think @barry-jin?

[fhieber]

Would it be more beneficial to have the full multihead attention primitive exposed in the API instead (it could still be implemented by the interleaved matmuls)? Both Keras and pyTorch have it as a layer, and this could make the usage easier. What do you think @barry-jin?

Can we have both? We'd like to upgrade Sockeye to MX2 without changing too many things at the same time.

[barry-jin]

@ptrendx Thanks for bringing it up. It's a good idea to have multihead attention layer in gluon and I will add it in a seperate PR.
@fhieber Yes, we can have both, a high level multihead attention layer and low level interleaved_matmul_* operators in npx name space.

[Zha0q1]

thanks for bring the interleaved matmul ops into npx! We have them in mx2onnx

[fhieber]

Can this be merged?