[5/N] pass the whole config to model

[youkaichao]

contains code from #9978 , need to merge that first.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀

[DarkLight1337]

I think you should not provide a default value for position_embedding so people won't forget about it.

Is it intended that you add prefix argument here? (Same question for the other models).

[youkaichao]

we need to unify the function signature, so that vllm_config (required) and prefix (optional) are enough to construct any model (as long as they have correct config).

this means we cannot have any other required parameters.

I think the position_embedding here should be fine, because it is only used for internal classes. Code outside of this file should not be aware of it.

[DarkLight1337]

I think prefix should only be an argument if we have actually implemented quantization support for that model. Is this necessary for the model construction code?

[DarkLight1337]

Also, this particular model is a base class and will not be called directly by the model builder. As long as the subclass doesn't have position_embedding in its argument list, it should not affect model builder code.

[youkaichao]

I think prefix should only be an argument if we have actually implemented quantization support for that model. Is this necessary for the model construction code?

we should keep this uniform signature, and if people want to add quantization support later, it is easier. functionality support like lora, quantization, etc. should be done via checking the config, rather than checking the function signature.

[DarkLight1337]

Can you add some helper functions to VllmConfig like raise_for_unsupported_quant so we have a standardized way of explicitly indicating that the model doesn't support quantization, and call it in the top-level models that don't originally support prefix?

[DarkLight1337]

Given this, I'm fine with the change.

[youkaichao]

I will ask neural magic folks to add this. I'm not familiar with how to tell if a model supports a quantization scheme.

[DarkLight1337]

cc @mgoin

[DarkLight1337]

As per offline discussion, let's address this in another PR.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. @youkaichao please rebase it. https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[DarkLight1337]

As per offline discussion, let's merge this first to unblock the next steps.