如何以 int4 的精度运行 glm-4-9b #211

ponponon · 2024-06-19T09:50:25Z

ponponon
Jun 19, 2024

如何以 int4 的精度运行 glm-4-9b

下面是我的代码：

from modelscope import AutoTokenizer, AutoModel, snapshot_download

model_dir = snapshot_download("ZhipuAI/glm-4-9b")
tokenizer = AutoTokenizer.from_pretrained(model_dir, trust_remote_code=True)
model = AutoModel.from_pretrained(
    model_dir, trust_remote_code=True).half().cuda()
model = model.eval()
response, history = model.chat(tokenizer, "你好", history=[])
print(response)
response, history = model.chat(tokenizer, "晚上睡不着应该怎么办", history=history)
print(response)

但是运行报错了

2024-06-19 09:20:09,600 - modelscope - INFO - PyTorch version 2.1.2 Found.
2024-06-19 09:20:09,601 - modelscope - INFO - Loading ast index from /home/pon/.cache/modelscope/ast_indexer
2024-06-19 09:20:09,638 - modelscope - INFO - Loading done! Current index file version is 1.10.0, with md5 53099a64fea000e3b62bcc73f188ebff and a total number of 946 components indexed
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:04<00:00,  2.17it/s]
Traceback (most recent call last):
  File "/home/pon/code/work/vt/track_judge/dev/modelscope_chatglm4.py", line 5, in <module>
    model = AutoModel.from_pretrained(model_dir, trust_remote_code=True).half().cuda()
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pon/.local/share/virtualenvs/track_judge-uFhqqWbT/lib/python3.11/site-packages/transformers/modeling_utils.py", line 2432, in cuda
    return super().cuda(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pon/.local/share/virtualenvs/track_judge-uFhqqWbT/lib/python3.11/site-packages/torch/nn/modules/module.py", line 918, in cuda
    return self._apply(lambda t: t.cuda(device))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pon/.local/share/virtualenvs/track_judge-uFhqqWbT/lib/python3.11/site-packages/torch/nn/modules/module.py", line 810, in _apply
    module._apply(fn)
  File "/home/pon/.local/share/virtualenvs/track_judge-uFhqqWbT/lib/python3.11/site-packages/torch/nn/modules/module.py", line 810, in _apply
    module._apply(fn)
  File "/home/pon/.local/share/virtualenvs/track_judge-uFhqqWbT/lib/python3.11/site-packages/torch/nn/modules/module.py", line 810, in _apply
    module._apply(fn)
  [Previous line repeated 3 more times]
  File "/home/pon/.local/share/virtualenvs/track_judge-uFhqqWbT/lib/python3.11/site-packages/torch/nn/modules/module.py", line 833, in _apply
    param_applied = fn(param)
                    ^^^^^^^^^
  File "/home/pon/.local/share/virtualenvs/track_judge-uFhqqWbT/lib/python3.11/site-packages/torch/nn/modules/module.py", line 918, in <lambda>
    return self._apply(lambda t: t.cuda(device))
                                 ^^^^^^^^^^^^^^
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 108.00 MiB. GPU 0 has a total capacty of 14.58 GiB of which 97.56 MiB is free. Including non-PyTorch memory, this process has 14.48 GiB memory in use. Of the allocated memory 14.38 GiB is allocated by PyTorch, and 1.15 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

因为我只有一张 Tesla T4 显卡，只有 16GB 的显存

─➤  nvidia-smi
Wed Jun 19 09:48:27 2024       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.161.08             Driver Version: 535.161.08   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla T4                       Off | 00000000:AF:00.0 Off |                    0 |
| N/A   45C    P0              27W /  70W |      2MiB / 15360MiB |      7%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+