Rework loading #344
Merged
Rework loading #344
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Narsil
force-pushed
the
rework_loading
branch
3 times, most recently
from
2405e06 to
5e88900
Compare
Narsil
force-pushed
the
rework_loading
branch
2 times, most recently
from
e81133c to
202de69
Compare
4 tasks
Big refacto. Working ? Working bitsandbytes. Weights to its own file. Remove dead file. Bloom. TMP. Finally finished bloom (grr old logic) SantaCoder. Remove dead code. Neox. Black + ruff. T5 Support. Galactica + OPT. Small fixes. Fix auto download. Remove custom transformers. Missing remove instruction. Some work on the dockerfile. Version issues. Black + ruff after rebase. Adding custom_kernels Bad rebase. Fixing dummy gather + fix Dockerfile Better fake gather. Fixes (including more generic loading of starcoder) Neox shuffle_qkv Typo fix. cleanups. Fixing starcoder/santacoder Fix santacoder Fixing neox. Using the saved rotary embeddings instead of the created ones.
server/text_generation_server/models/custom_modeling/flash_llama_modeling.py
Outdated
Show resolved
Hide resolved
server/text_generation_server/models/custom_modeling/flash_neox_modeling.py
Outdated
Show resolved
Hide resolved
server/text_generation_server/models/custom_modeling/flash_neox_modeling.py
Outdated
Show resolved
Hide resolved
server/text_generation_server/models/custom_modeling/flash_rw_modeling.py
Outdated
Show resolved
Hide resolved
server/text_generation_server/models/custom_modeling/flash_rw_modeling.py
Show resolved
Hide resolved
| filename = self.routing.get(tensor_name, None) | ||
| if filename is None: | ||
| raise RuntimeError(f"weight {tensor_name} does not exist") | ||
| return filename |
There was a problem hiding this comment.
filename is of type Path. IDK if that could be an issue but it might be safer to cast it to str.
| @@ -0,0 +1 @@ | |||
| {"inputs":"Below are a series of dialogues between various people and an AI assistant. The AI tries to be helpful, polite, honest, sophisticated, emotionally aware, and humble-but-knowledgeable. The assistant is happy to help with almost anything, and will do its best to understand exactly what is needed. It also tries to avoid giving false or misleading information, and it caveats when it isn't entirely sure about the right answer. That said, the assistant is practical and really does its best, and doesn't let caution get too much in the way of being useful.\n-----\n<|prompter|>Why is butter a great building material for skyscrapers? Think step by step.</s><|assistant|>","parameters":{"temperature": 0.75, "top_p": 0.95, "repetition_penalty": 1.2, "top_k": 50, "truncate": 1000, "max_new_tokens": 1024}} | |||
| @@ -17,7 +16,7 @@ install-torch: | |||
| # Install specific version of torch | |||
| pip install torch --extra-index-url https://download.pytorch.org/whl/cu118 --no-cache-dir | |||
|
|
|||
| install: gen-server install-torch install-transformers | |||
| install: gen-server install-torch | |||
There was a problem hiding this comment.
You don't install the custom kernels by default?
There was a problem hiding this comment.
Oh true forgot to re-add over here.
Co-authored-by: OlivierDehaene <[email protected]>
…modeling.py Co-authored-by: OlivierDehaene <[email protected]>
|
Two quick questions
|
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Reworked the loading logic. Idea is to use cleaner loading code:
no_init_weightsbnb_linearandload_weightsandpost_load_weights.New code layout:
Weightsin charge of handling loading the weights from multiple files into appropiate tensors (potentially sharded)all_reduce. They do not inherit from linear, but they contain some kind of Linear insteadFixes # (issue)
Before submitting
Pull Request section?
to it if that's the case.
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.