[Misc][Quark] Upstream Quark format to VLLM

[kewang-xlnx]

This PR is related to the issue discussed in this #10294.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @kewang-xlnx.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[robertgshaw2-redhat]

Its a bit unfortunate that:

• We will have all this duplicated code
• We will have multiple model formats to maintain

However, I understand that AMD wants to push forward with Quark, so I think this approach is very reasonable. I would like to take on this effort myself and do two things:

• Factor out some of the utility functions such that we do not have too much copy / paste code between the backends
• Leverage the Kernel abstraction such that compressed-tensors models can run on RoCM backend

To do this, it would be very helpful if the Quark team could provide:

• A sample W8A8-int8 model
• A sample W8A8-fp8 model

If this is done, I can take on the effort of finalizing the integration. Thanks!

[kewang-xlnx]

Its a bit unfortunate that:

• We will have all this duplicated code
• We will have multiple model formats to maintain

However, I understand that AMD wants to push forward with Quark, so I think this approach is very reasonable. I would like to take on this effort myself and do two things:

• Factor out some of the utility functions such that we do not have too much copy / paste code between the backends
• Leverage the Kernel abstraction such that compressed-tensors models can run on RoCM backend

To do this, it would be very helpful if the Quark team could provide:

• A sample W8A8-int8 model
• A sample W8A8-fp8 model

If this is done, I can take on the effort of finalizing the integration. Thanks!

Hi @robertgshaw2-neuralmagic. Thank you for your comments.

Your proposed work is highly valuable and will benefit both VLLM contributors and users. Factoring out utility functions to reduce code duplication and leveraging the Kernel abstraction for compatibility with the RoCM backend are excellent steps forward.

Here is a config file of W8A8-fp8:
config.json
And this is a config file of W8A8-int8:
config.json

key names and data types of Quark safetensors

model.layers.1.self_attn.k_proj.input_scale, 	torch.float16
model.layers.1.self_attn.k_proj.weight, 	torch.float8_e4m3fn
model.layers.1.self_attn.k_proj.weight_scale, 	torch.float16
model.layers.1.self_attn.o_proj.input_scale, 	torch.float16
model.layers.1.self_attn.o_proj.weight, 	torch.float8_e4m3fn
model.layers.1.self_attn.o_proj.weight_scale, 	torch.float16
model.layers.1.self_attn.q_proj.input_scale, 	torch.float16
model.layers.1.self_attn.q_proj.weight, 	torch.float8_e4m3fn
model.layers.1.self_attn.q_proj.weight_scale, 	torch.float16
model.layers.1.self_attn.v_proj.input_scale, 	torch.float16
model.layers.1.self_attn.v_proj.weight, 	torch.float8_e4m3fn
model.layers.1.self_attn.v_proj.weight_scale, 	torch.float16

KV scale format if kv cache used

model.layers.1.self_attn.k_proj.output_scale, 	torch.float16
model.layers.1.self_attn.v_proj.output_scale, 	torch.float16

And it’s important that the integration of the Quark format into VLLM is a priority. To ensure long-term maintainability, we would like to keep the responsibility for maintaining the Quark format within AMD’s team. Our team will closely monitor the integration to minimize any maintenance overhead for the vLLM project.

Let us know if you need any further support during the process. Thanks again for your collaboration!

[mgoin]

KV scale format if kv cache used

model.layers.1.self_attn.k_proj.output_scale, 	torch.float16
model.layers.1.self_attn.v_proj.output_scale, 	torch.float16

Hi @kewang-xlnx can you describe what happens in the case of a model with fused QKV proj, such as Phi-3? In several cases models do not have separate k_proj/v_proj linear layers with which to register output scales on. This is why we register explicit self_attn.k_scale and self_attn.v_scale in compressed-tensors. We could likely reuse the same utility if you choose the same place to register these (on the Attention module)

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @kewang-xlnx.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[kewang-xlnx]

KV scale format if kv cache used

model.layers.1.self_attn.k_proj.output_scale, 	torch.float16
model.layers.1.self_attn.v_proj.output_scale, 	torch.float16

Hi @kewang-xlnx can you describe what happens in the case of a model with fused QKV proj, such as Phi-3? In several cases models do not have separate k_proj/v_proj linear layers with which to register output scales on. This is why we register explicit self_attn.k_scale and self_attn.v_scale in compressed-tensors. We could likely reuse the same utility if you choose the same place to register these (on the Attention module)

Hi @mgoin. If the kv_scale of fused QKV proj in quark is as below, take Phi-3 for example.
model.layers.22.self_attn.qkv_proj.output_scale
This is why we need to remap one kv_scale to both attn.k_scale and attn.v_scale.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @kewang-xlnx.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[mgoin]

Apologies for the delay and thanks for the changes! This seems to be in a decent place to land. Would you have a small quark dense and/or moe model we could add to CI @kewang-xlnx ?

[mgoin]

Thank you for your hard work @kewang-xlnx!