Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pytorch converted model returns .GGML_ASSERT: ggml-cuda.cu:6115: false #4017

Closed
4 tasks done
rvandernoort opened this issue Nov 10, 2023 · 1 comment
Closed
4 tasks done

Pytorch converted model returns .GGML_ASSERT: ggml-cuda.cu:6115: false #4017

rvandernoort opened this issue Nov 10, 2023 · 1 comment
Labels
bug-unconfirmed

Comments

@rvandernoort
Copy link

Prerequisites

Please answer the following questions for yourself before submitting an issue.

Expected Behavior

Please provide a detailed written description of what you were trying to do, and what you expected llama.cpp to do.

Hi just started working with llama.cpp and I stumbled on this issue. Maybe its my side but I'm not really getting it working. I want to use the full 32 bit GGUF model converted from a pytorch model if possible without any more quantization. Can you help me or is this a bug? If any more information is required, let me know

  1. Convert model to 32 bit GGUF with python3 convert.py ./models/tinyllama-1.1b-chat-v0.3
  2. Run this model using llama.cpp docker image

Current Behavior

Please provide a detailed written description of what llama.cpp did, instead.

  1. Convert model to 32 bit GGUF with python3 convert.py ./models/tinyllama-1.1b-chat-v0.3 (succeeds)
  2. Run this model using llama.cpp docker image (fails)

while other bit level does work:

  1. Concert model to 16 bit GGUF with python3 convert.py ./models/tinyllama-1.1b-chat-v0.3 --outtype f16 (succeeds)
  2. Run this model using llama.cpp docker image (succeeds)

Environment and Context

Please provide detailed information about your computer setup. This is important in case the issue is not reproducible except for under certain specific conditions.

  • Physical (or virtual) hardware you are using, e.g. for Linux:

$ lscpu

Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         48 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  24
  On-line CPU(s) list:   0-23
Vendor ID:               AuthenticAMD
  Model name:            AMD Ryzen 9 7900X 12-Core Processor
    CPU family:          25
    Model:               97
    Thread(s) per core:  2
    Core(s) per socket:  12
    Socket(s):           1
    Stepping:            2
    Frequency boost:     enabled
    CPU max MHz:         5732,7139
    CPU min MHz:         3000,0000
    BogoMIPS:            9381.89
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc
                          rep_good amd_lbr_v2 nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f
                         16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr
                         _llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba perfmon_v2 ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdsee
                         d adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local avx51
                         2_bf16 clzero irperf xsaveerptr rdpru wbnoinvd cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold av
                         ic v_vmsave_vmload vgif x2avic v_spec_ctrl avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid overfl
                         ow_recov succor smca fsrm flush_l1d
Virtualization features: 
  Virtualization:        AMD-V
Caches (sum of all):     
  L1d:                   384 KiB (12 instances)
  L1i:                   384 KiB (12 instances)
  L2:                    12 MiB (12 instances)
  L3:                    64 MiB (2 instances)
NUMA:                    
  NUMA node(s):          1
  NUMA node0 CPU(s):     0-23

NVIDIA 4090
  • Operating System, e.g. for Linux:

$ uname -a

Linux GreenServer 6.2.0-34-generic #34~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Sep  7 13:12:03 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
  • SDK version, e.g. for Linux:
$ python3 --version
Python 3.10.12
$ make --version
GNU Make 4.3
$ g++ --version
g++ (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0

Contaierfile: 
FROM ghcr.io/ggerganov/llama.cpp:full-cuda
ENTRYPOINT ['./main']

Docker compose file:
services:
    llama.cpp:
        image: llama.cpp
        container_name: llama.cpp-gpu
        build:
            context: '${PWD}/'
            dockerfile: '${PWD}/Containerfile.orig'
        volumes:
            - '${PWD}/models:/models'
        command: -m /models/TinyLLama/original/ggml-model-f32.gguf -p "Building a website can be done in 10 simple steps:" -n 1024 --seed 12345678 -t 2 --n-gpu-layers 99
        deploy:
            resources:
                reservations:
                    devices:
                        - driver: nvidia
                          count: 1
                          capabilities: [gpu]

Failure Information (for bugs)

Please help provide information about the failure / bug.

.GGML_ASSERT: ggml-cuda.cu:6115: false

Its unclear for me what this bug means.

Steps to Reproduce

Please provide detailed steps for reproducing the issue. We are not sitting in front of your screen, so the more detail the better.

  1. download pytorch_model.bin from https://huggingface.co/PY007/TinyLlama-1.1B-Chat-v0.3 in ./models/tinyllama-1.1b-chat-v0.3
  2. python3 convert.py ./models/tinyllama-1.1b-chat-v0.3
  3. Run docker compose up

Failure Logs

Please include any relevant log snippets or files. If it works under one configuration but not under another, please provide logs for both configurations and their corresponding outputs so it is easy to see where behavior changes.

Log of 32 bit container

[+] Running 1/0
 ✔ Container llama.cpp-gpu  Created                                                                                                                                               0.0s 
Attaching to llama.cpp-gpu
llama.cpp-gpu  | Log start
llama.cpp-gpu  | main: build = 0 (unknown)
llama.cpp-gpu  | main: built with cc (Ubuntu 11.3.0-1ubuntu1~22.04.1) 11.3.0 for x86_64-linux-gnu
llama.cpp-gpu  | main: seed  = 12345678
llama.cpp-gpu  | ggml_init_cublas: found 1 CUDA devices:
llama.cpp-gpu  |   Device 0: NVIDIA GeForce RTX 4090, compute capability 8.9
llama.cpp-gpu  | llama_model_loader: loaded meta data with 20 key-value pairs and 201 tensors from /models/TinyLLama/original/TinyLlama-1.1B-Chat-v0.3/ggml-model-f32.gguf (version unknown)
llama.cpp-gpu  | llama_model_loader: - tensor    0:                token_embd.weight f32      [  2048, 32003,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor    1:              blk.0.attn_q.weight f32      [  2048,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor    2:              blk.0.attn_k.weight f32      [  2048,   256,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor    3:              blk.0.attn_v.weight f32      [  2048,   256,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor    4:         blk.0.attn_output.weight f32      [  2048,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor    5:            blk.0.ffn_gate.weight f32      [  2048,  5632,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor    6:              blk.0.ffn_up.weight f32      [  2048,  5632,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor    7:            blk.0.ffn_down.weight f32      [  5632,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor    8:           blk.0.attn_norm.weight f32      [  2048,     1,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor    9:            blk.0.ffn_norm.weight f32      [  2048,     1,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   10:              blk.1.attn_q.weight f32      [  2048,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   11:              blk.1.attn_k.weight f32      [  2048,   256,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   12:              blk.1.attn_v.weight f32      [  2048,   256,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   13:         blk.1.attn_output.weight f32      [  2048,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   14:            blk.1.ffn_gate.weight f32      [  2048,  5632,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   15:              blk.1.ffn_up.weight f32      [  2048,  5632,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   16:            blk.1.ffn_down.weight f32      [  5632,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   17:           blk.1.attn_norm.weight f32      [  2048,     1,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   18:            blk.1.ffn_norm.weight f32      [  2048,     1,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   19:              blk.2.attn_q.weight f32      [  2048,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   20:              blk.2.attn_k.weight f32      [  2048,   256,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   21:              blk.2.attn_v.weight f32      [  2048,   256,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   22:         blk.2.attn_output.weight f32      [  2048,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   23:            blk.2.ffn_gate.weight f32      [  2048,  5632,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   24:              blk.2.ffn_up.weight f32      [  2048,  5632,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   25:            blk.2.ffn_down.weight f32      [  5632,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   26:           blk.2.attn_norm.weight f32      [  2048,     1,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   27:            blk.2.ffn_norm.weight f32      [  2048,     1,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   28:              blk.3.attn_q.weight f32      [  2048,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   29:              blk.3.attn_k.weight f32      [  2048,   256,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   30:              blk.3.attn_v.weight f32      [  2048,   256,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   31:         blk.3.attn_output.weight f32      [  2048,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   32:            blk.3.ffn_gate.weight f32      [  2048,  5632,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   33:              blk.3.ffn_up.weight f32      [  2048,  5632,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   34:            blk.3.ffn_down.weight f32      [  5632,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   35:           blk.3.attn_norm.weight f32      [  2048,     1,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   36:            blk.3.ffn_norm.weight f32      [  2048,     1,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   37:              blk.4.attn_q.weight f32      [  2048,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   38:              blk.4.attn_k.weight f32      [  2048,   256,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   39:              blk.4.attn_v.weight f32      [  2048,   256,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   40:         blk.4.attn_output.weight f32      [  2048,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   41:            blk.4.ffn_gate.weight f32      [  2048,  5632,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   42:              blk.4.ffn_up.weight f32      [  2048,  5632,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   43:            blk.4.ffn_down.weight f32      [  5632,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   44:           blk.4.attn_norm.weight f32      [  2048,     1,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   45:            blk.4.ffn_norm.weight f32      [  2048,     1,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   46:              blk.5.attn_q.weight f32      [  2048,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   47:              blk.5.attn_k.weight f32      [  2048,   256,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   48:              blk.5.attn_v.weight f32      [  2048,   256,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   49:         blk.5.attn_output.weight f32      [  2048,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   50:            blk.5.ffn_gate.weight f32      [  2048,  5632,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   51:              blk.5.ffn_up.weight f32      [  2048,  5632,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   52:            blk.5.ffn_down.weight f32      [  5632,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   53:           blk.5.attn_norm.weight f32      [  2048,     1,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   54:            blk.5.ffn_norm.weight f32      [  2048,     1,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   55:              blk.6.attn_q.weight f32      [  2048,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   56:              blk.6.attn_k.weight f32      [  2048,   256,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   57:              blk.6.attn_v.weight f32      [  2048,   256,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   58:         blk.6.attn_output.weight f32      [  2048,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   59:            blk.6.ffn_gate.weight f32      [  2048,  5632,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   60:              blk.6.ffn_up.weight f32      [  2048,  5632,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   61:            blk.6.ffn_down.weight f32      [  5632,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   62:           blk.6.attn_norm.weight f32      [  2048,     1,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   63:            blk.6.ffn_norm.weight f32      [  2048,     1,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   64:              blk.7.attn_q.weight f32      [  2048,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   65:              blk.7.attn_k.weight f32      [  2048,   256,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   66:              blk.7.attn_v.weight f32      [  2048,   256,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   67:         blk.7.attn_output.weight f32      [  2048,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   68:            blk.7.ffn_gate.weight f32      [  2048,  5632,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   69:              blk.7.ffn_up.weight f32      [  2048,  5632,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   70:            blk.7.ffn_down.weight f32      [  5632,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   71:           blk.7.attn_norm.weight f32      [  2048,     1,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   72:            blk.7.ffn_norm.weight f32      [  2048,     1,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   73:              blk.8.attn_q.weight f32      [  2048,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   74:              blk.8.attn_k.weight f32      [  2048,   256,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   75:              blk.8.attn_v.weight f32      [  2048,   256,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   76:         blk.8.attn_output.weight f32      [  2048,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   77:            blk.8.ffn_gate.weight f32      [  2048,  5632,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   78:              blk.8.ffn_up.weight f32      [  2048,  5632,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   79:            blk.8.ffn_down.weight f32      [  5632,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   80:           blk.8.attn_norm.weight f32      [  2048,     1,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   81:            blk.8.ffn_norm.weight f32      [  2048,     1,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   82:              blk.9.attn_q.weight f32      [  2048,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   83:              blk.9.attn_k.weight f32      [  2048,   256,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   84:              blk.9.attn_v.weight f32      [  2048,   256,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   85:         blk.9.attn_output.weight f32      [  2048,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   86:            blk.9.ffn_gate.weight f32      [  2048,  5632,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   87:              blk.9.ffn_up.weight f32      [  2048,  5632,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   88:            blk.9.ffn_down.weight f32      [  5632,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   89:           blk.9.attn_norm.weight f32      [  2048,     1,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   90:            blk.9.ffn_norm.weight f32      [  2048,     1,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   91:             blk.10.attn_q.weight f32      [  2048,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   92:             blk.10.attn_k.weight f32      [  2048,   256,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   93:             blk.10.attn_v.weight f32      [  2048,   256,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   94:        blk.10.attn_output.weight f32      [  2048,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   95:           blk.10.ffn_gate.weight f32      [  2048,  5632,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   96:             blk.10.ffn_up.weight f32      [  2048,  5632,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   97:           blk.10.ffn_down.weight f32      [  5632,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   98:          blk.10.attn_norm.weight f32      [  2048,     1,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor   99:           blk.10.ffn_norm.weight f32      [  2048,     1,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  100:             blk.11.attn_q.weight f32      [  2048,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  101:             blk.11.attn_k.weight f32      [  2048,   256,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  102:             blk.11.attn_v.weight f32      [  2048,   256,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  103:        blk.11.attn_output.weight f32      [  2048,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  104:           blk.11.ffn_gate.weight f32      [  2048,  5632,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  105:             blk.11.ffn_up.weight f32      [  2048,  5632,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  106:           blk.11.ffn_down.weight f32      [  5632,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  107:          blk.11.attn_norm.weight f32      [  2048,     1,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  108:           blk.11.ffn_norm.weight f32      [  2048,     1,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  109:             blk.12.attn_q.weight f32      [  2048,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  110:             blk.12.attn_k.weight f32      [  2048,   256,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  111:             blk.12.attn_v.weight f32      [  2048,   256,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  112:        blk.12.attn_output.weight f32      [  2048,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  113:           blk.12.ffn_gate.weight f32      [  2048,  5632,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  114:             blk.12.ffn_up.weight f32      [  2048,  5632,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  115:           blk.12.ffn_down.weight f32      [  5632,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  116:          blk.12.attn_norm.weight f32      [  2048,     1,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  117:           blk.12.ffn_norm.weight f32      [  2048,     1,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  118:             blk.13.attn_q.weight f32      [  2048,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  119:             blk.13.attn_k.weight f32      [  2048,   256,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  120:             blk.13.attn_v.weight f32      [  2048,   256,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  121:        blk.13.attn_output.weight f32      [  2048,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  122:           blk.13.ffn_gate.weight f32      [  2048,  5632,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  123:             blk.13.ffn_up.weight f32      [  2048,  5632,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  124:           blk.13.ffn_down.weight f32      [  5632,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  125:          blk.13.attn_norm.weight f32      [  2048,     1,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  126:           blk.13.ffn_norm.weight f32      [  2048,     1,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  127:             blk.14.attn_q.weight f32      [  2048,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  128:             blk.14.attn_k.weight f32      [  2048,   256,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  129:             blk.14.attn_v.weight f32      [  2048,   256,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  130:        blk.14.attn_output.weight f32      [  2048,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  131:           blk.14.ffn_gate.weight f32      [  2048,  5632,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  132:             blk.14.ffn_up.weight f32      [  2048,  5632,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  133:           blk.14.ffn_down.weight f32      [  5632,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  134:          blk.14.attn_norm.weight f32      [  2048,     1,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  135:           blk.14.ffn_norm.weight f32      [  2048,     1,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  136:             blk.15.attn_q.weight f32      [  2048,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  137:             blk.15.attn_k.weight f32      [  2048,   256,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  138:             blk.15.attn_v.weight f32      [  2048,   256,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  139:        blk.15.attn_output.weight f32      [  2048,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  140:           blk.15.ffn_gate.weight f32      [  2048,  5632,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  141:             blk.15.ffn_up.weight f32      [  2048,  5632,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  142:           blk.15.ffn_down.weight f32      [  5632,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  143:          blk.15.attn_norm.weight f32      [  2048,     1,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  144:           blk.15.ffn_norm.weight f32      [  2048,     1,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  145:             blk.16.attn_q.weight f32      [  2048,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  146:             blk.16.attn_k.weight f32      [  2048,   256,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  147:             blk.16.attn_v.weight f32      [  2048,   256,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  148:        blk.16.attn_output.weight f32      [  2048,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  149:           blk.16.ffn_gate.weight f32      [  2048,  5632,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  150:             blk.16.ffn_up.weight f32      [  2048,  5632,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  151:           blk.16.ffn_down.weight f32      [  5632,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  152:          blk.16.attn_norm.weight f32      [  2048,     1,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  153:           blk.16.ffn_norm.weight f32      [  2048,     1,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  154:             blk.17.attn_q.weight f32      [  2048,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  155:             blk.17.attn_k.weight f32      [  2048,   256,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  156:             blk.17.attn_v.weight f32      [  2048,   256,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  157:        blk.17.attn_output.weight f32      [  2048,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  158:           blk.17.ffn_gate.weight f32      [  2048,  5632,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  159:             blk.17.ffn_up.weight f32      [  2048,  5632,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  160:           blk.17.ffn_down.weight f32      [  5632,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  161:          blk.17.attn_norm.weight f32      [  2048,     1,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  162:           blk.17.ffn_norm.weight f32      [  2048,     1,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  163:             blk.18.attn_q.weight f32      [  2048,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  164:             blk.18.attn_k.weight f32      [  2048,   256,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  165:             blk.18.attn_v.weight f32      [  2048,   256,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  166:        blk.18.attn_output.weight f32      [  2048,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  167:           blk.18.ffn_gate.weight f32      [  2048,  5632,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  168:             blk.18.ffn_up.weight f32      [  2048,  5632,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  169:           blk.18.ffn_down.weight f32      [  5632,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  170:          blk.18.attn_norm.weight f32      [  2048,     1,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  171:           blk.18.ffn_norm.weight f32      [  2048,     1,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  172:             blk.19.attn_q.weight f32      [  2048,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  173:             blk.19.attn_k.weight f32      [  2048,   256,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  174:             blk.19.attn_v.weight f32      [  2048,   256,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  175:        blk.19.attn_output.weight f32      [  2048,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  176:           blk.19.ffn_gate.weight f32      [  2048,  5632,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  177:             blk.19.ffn_up.weight f32      [  2048,  5632,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  178:           blk.19.ffn_down.weight f32      [  5632,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  179:          blk.19.attn_norm.weight f32      [  2048,     1,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  180:           blk.19.ffn_norm.weight f32      [  2048,     1,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  181:             blk.20.attn_q.weight f32      [  2048,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  182:             blk.20.attn_k.weight f32      [  2048,   256,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  183:             blk.20.attn_v.weight f32      [  2048,   256,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  184:        blk.20.attn_output.weight f32      [  2048,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  185:           blk.20.ffn_gate.weight f32      [  2048,  5632,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  186:             blk.20.ffn_up.weight f32      [  2048,  5632,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  187:           blk.20.ffn_down.weight f32      [  5632,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  188:          blk.20.attn_norm.weight f32      [  2048,     1,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  189:           blk.20.ffn_norm.weight f32      [  2048,     1,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  190:             blk.21.attn_q.weight f32      [  2048,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  191:             blk.21.attn_k.weight f32      [  2048,   256,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  192:             blk.21.attn_v.weight f32      [  2048,   256,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  193:        blk.21.attn_output.weight f32      [  2048,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  194:           blk.21.ffn_gate.weight f32      [  2048,  5632,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  195:             blk.21.ffn_up.weight f32      [  2048,  5632,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  196:           blk.21.ffn_down.weight f32      [  5632,  2048,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  197:          blk.21.attn_norm.weight f32      [  2048,     1,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  198:           blk.21.ffn_norm.weight f32      [  2048,     1,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  199:               output_norm.weight f32      [  2048,     1,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - tensor  200:                    output.weight f32      [  2048, 32003,     1,     1 ]
llama.cpp-gpu  | llama_model_loader: - kv   0:                       general.architecture str     
llama.cpp-gpu  | llama_model_loader: - kv   1:                               general.name str     
llama.cpp-gpu  | llama_model_loader: - kv   2:                       llama.context_length u32     
llama.cpp-gpu  | llama_model_loader: - kv   3:                     llama.embedding_length u32     
llama.cpp-gpu  | llama_model_loader: - kv   4:                          llama.block_count u32     
llama.cpp-gpu  | llama_model_loader: - kv   5:                  llama.feed_forward_length u32     
llama.cpp-gpu  | llama_model_loader: - kv   6:                 llama.rope.dimension_count u32     
llama.cpp-gpu  | llama_model_loader: - kv   7:                 llama.attention.head_count u32     
llama.cpp-gpu  | llama_model_loader: - kv   8:              llama.attention.head_count_kv u32     
llama.cpp-gpu  | llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32     
llama.cpp-gpu  | llama_model_loader: - kv  10:                       llama.rope.freq_base f32     
llama.cpp-gpu  | llama_model_loader: - kv  11:                          general.file_type u32     
llama.cpp-gpu  | llama_model_loader: - kv  12:                       tokenizer.ggml.model str     
llama.cpp-gpu  | llama_model_loader: - kv  13:                      tokenizer.ggml.tokens arr     
llama.cpp-gpu  | llama_model_loader: - kv  14:                      tokenizer.ggml.scores arr     
llama.cpp-gpu  | llama_model_loader: - kv  15:                  tokenizer.ggml.token_type arr     
llama.cpp-gpu  | llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32     
llama.cpp-gpu  | llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32     
llama.cpp-gpu  | llama_model_loader: - kv  18:            tokenizer.ggml.unknown_token_id u32     
llama.cpp-gpu  | llama_model_loader: - kv  19:            tokenizer.ggml.padding_token_id u32     
llama.cpp-gpu  | llama_model_loader: - type  f32:  201 tensors
llama.cpp-gpu  | llm_load_print_meta: format           = unknown
llama.cpp-gpu  | llm_load_print_meta: arch             = llama
llama.cpp-gpu  | llm_load_print_meta: vocab type       = SPM
llama.cpp-gpu  | llm_load_print_meta: n_vocab          = 32003
llama.cpp-gpu  | llm_load_print_meta: n_merges         = 0
llama.cpp-gpu  | llm_load_print_meta: n_ctx_train      = 2048
llama.cpp-gpu  | llm_load_print_meta: n_embd           = 2048
llama.cpp-gpu  | llm_load_print_meta: n_head           = 32
llama.cpp-gpu  | llm_load_print_meta: n_head_kv        = 4
llama.cpp-gpu  | llm_load_print_meta: n_layer          = 22
llama.cpp-gpu  | llm_load_print_meta: n_rot            = 64
llama.cpp-gpu  | llm_load_print_meta: n_gqa            = 8
llama.cpp-gpu  | llm_load_print_meta: f_norm_eps       = 0.0e+00
llama.cpp-gpu  | llm_load_print_meta: f_norm_rms_eps   = 1.0e-05
llama.cpp-gpu  | llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llama.cpp-gpu  | llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llama.cpp-gpu  | llm_load_print_meta: n_ff             = 5632
llama.cpp-gpu  | llm_load_print_meta: freq_base_train  = 10000.0
llama.cpp-gpu  | llm_load_print_meta: freq_scale_train = 1
llama.cpp-gpu  | llm_load_print_meta: model type       = ?B
llama.cpp-gpu  | llm_load_print_meta: model ftype      = all F32
llama.cpp-gpu  | llm_load_print_meta: model params     = 1.10 B
llama.cpp-gpu  | llm_load_print_meta: model size       = 4.10 GiB (32.00 BPW) 
llama.cpp-gpu  | llm_load_print_meta: general.name   = models
llama.cpp-gpu  | llm_load_print_meta: BOS token = 1 '<s>'
llama.cpp-gpu  | llm_load_print_meta: EOS token = 2 '</s>'
llama.cpp-gpu  | llm_load_print_meta: UNK token = 0 '<unk>'
llama.cpp-gpu  | llm_load_print_meta: PAD token = 32000 '[PAD]'
llama.cpp-gpu  | llm_load_print_meta: LF token  = 13 '<0x0A>'
llama.cpp-gpu  | llm_load_tensors: ggml ctx size =    0.07 MB
llama.cpp-gpu  | llm_load_tensors: using CUDA for GPU acceleration
llama.cpp-gpu  | llm_load_tensors: mem required  =  250.09 MB
llama.cpp-gpu  | llm_load_tensors: offloading 22 repeating layers to GPU
llama.cpp-gpu  | llm_load_tensors: offloading non-repeating layers to GPU
llama.cpp-gpu  | llm_load_tensors: offloaded 25/25 layers to GPU
llama.cpp-gpu  | llm_load_tensors: VRAM used: 3946.38 MB
llama.cpp-gpu  | .GGML_ASSERT: ggml-cuda.cu:6115: false
llama.cpp-gpu exited with code 139
@AndrewGodfrey
Copy link
Contributor

I hit this too. It seems to just be a missing case in a switch statement.

AndrewGodfrey added a commit to AndrewGodfrey/llama.cpp that referenced this issue Nov 16, 2023
@AndrewGodfrey
Fix ggerganov#4017
e77094f
@AndrewGodfrey AndrewGodfrey mentioned this issue Nov 16, 2023
Merged
olexiyb pushed a commit to Sanctum-AI/llama.cpp that referenced this issue Nov 23, 2023
@AndrewGodfrey @cebtenzzre @olexiyb
cuda : get_row_rounding F32 (ggerganov#4095)
11649d4 
* Fix ggerganov#4017

* Update ggml-cuda.cu

Co-authored-by: Jared Van Bortel <[email protected]>

* Update ggml-cuda.cu

Co-authored-by: Jared Van Bortel <[email protected]>

---------

Co-authored-by: Jared Van Bortel <[email protected]>
hodlen added a commit to hodlen/llama.cpp that referenced this issue Apr 1, 2024
@hodlen @ggerganov @Mrkvak
Squash commits until Mixtral support for merging
25c80b9 
llama : restore prefix space in llama tokenizer (ggerganov#4081)

gguf : fix potential infinite loops while parsing (ggerganov#4100)

Co-authored-by: Bernhard Gstrein <[email protected]>

Respect tokenizer.ggml.add_bos_token value when tokenizing (ggerganov#4040)

* gguf-py: gguf-dump: Respect --no-tensor flag in JSON mode.

* Respect add_bos_token GGUF metadata value

* gguf-py: Try to fix SpecialVocab giving up too easily for the Nth time

llama : fix data units (ggerganov#4101)

* llama : fix data units

ggml-ci

* Revert "llama : fix data units"

This reverts commit f5feac8.

* llama : disambiguate data units

ggml-ci

cuda : get_row_rounding F32 (ggerganov#4095)

* Fix ggerganov#4017

* Update ggml-cuda.cu

Co-authored-by: Jared Van Bortel <[email protected]>

* Update ggml-cuda.cu

Co-authored-by: Jared Van Bortel <[email protected]>

---------

Co-authored-by: Jared Van Bortel <[email protected]>

finetune : zero the loraB initial vectors (ggerganov#4082)

* finetune : zero the loraB initial vectors

Without this, the first iteration is starting out far from the base model, instead of exactly on it.
Zeroing loraB is what the paper recommends. loralib also zeroes at least one of the init vector pairs
(though it departs from the paper in using a different distribution for the other vector, in some cases).

* tabs to spaces

* Use ggml_set_zero instead of adding a new function

finetune : speed-up ggml_compute_forward_out_prod_f32 via BLAS (ggerganov#4079)

* Remove logically superfluous assertions and order by dimension

* Use cblas_sgemm() to implement ggml_compute_forward_out_prod()

* Remove ggml_compute_forward_out_prod_use_blas(), fix compiling errors on cmake/zig, remove trailing whitespace

* Add openBLAS support for sgemm() in compute_forward_out_prod()

llama : add functions to get the model's metadata (ggerganov#4013)

* llama : add functions to get the model's metadata

* format -> std::to_string

* better documentation

train : move number of gpu layers argument parsing to common/train.cpp (ggerganov#4074)

- introduces help entry for the argument
 - cuts '--gpu-layers' form in order to simplify usage and documentation.

Signed-off-by: Jiri Podivin <[email protected]>
Co-authored-by: Jiri Podivin <[email protected]>

py : remove superfluous import statements (ggerganov#4076)

Signed-off-by: Jiri Podivin <[email protected]>
Co-authored-by: Jiri Podivin <[email protected]>

llava : fix compilation warning that fread return value is not used (ggerganov#4069)

common : improve yaml log escaping (ggerganov#4080)

* logging: improve escaping in yaml output

* logging: include review feedback

py : Falcon HF compatibility (ggerganov#4104)

Falcon HF compatibility

convert : use 'model' value if it exists. This allows karpathy/tinyllamas to load (ggerganov#4089)

Co-authored-by: Don Mahurin <@>

examples : add tokenize (ggerganov#4039)

tokenize : fix trailing whitespace

build : support ppc64le build for make and CMake (ggerganov#3963)

* build: support ppc64le build for make and CMake

* build: keep __POWER9_VECTOR__ ifdef and extend with __powerpc64__

Co-authored-by: Georgi Gerganov <[email protected]>

---------

Co-authored-by: Georgi Gerganov <[email protected]>

llama : increase max nodes (ggerganov#4115)

Clean up ggml-cuda.cu warnings when compiling with clang (for ROCM) (ggerganov#4124)

* ggml-cuda.cu: Clean up warnings when compiling with clang

* ggml-cuda.cu: Move static items into anonymous namespace

* ggml-cuda.cu: Fix use of namespace start macro

* Revert "ggml-cuda.cu: Fix use of namespace start macro"

This reverts commit 26c1149.

* Revert "ggml-cuda.cu: Move static items into anonymous namespace"

This reverts commit e29757e.

scripts : Remove missed baichuan convert script (ggerganov#4127)

tokenize example: Respect normal add BOS token behavior (ggerganov#4126)

Allow building with Makefile

gguf-py : export chat templates (ggerganov#4125)

* gguf-py : export chat templates

* llama.cpp : escape new lines in gguf kv info prints

* gguf-py : bump version

* gguf-py : check chat_template type

* gguf-py : initialize chat_template

gitignore : tokenize

common : comma should be semicolon (ggerganov#4137)

server : relay error messages (ggerganov#4131)

finetune : add --n-gpu-layers flag info to --help (ggerganov#4128)

Revert "finetune : add --n-gpu-layers flag info to --help (ggerganov#4128)"

This reverts commit 05e8301.

speculative : fix prompt tokenization in speculative example (ggerganov#4025)

* Support special tokens and not adding BOS to prompt in speculative

* Adapt to new should_add_bos function

* Ensure tgt and dft have same add_bos setting

ci : add flake8 to github actions (python linting) (ggerganov#4129)

Disabled rules:

* E203 Whitespace before ':' - disabled because we often use 'C' Style where values are aligned

* E211 Whitespace before '(' (E211) - disabled because we often use 'C' Style where values are aligned

* E221 Multiple spaces before operator - disabled because we often use 'C' Style where values are aligned

* E225 Missing whitespace around operator - disabled because it's broken so often it seems like a standard

* E231 Missing whitespace after ',', ';', or ':' - disabled because we often use 'C' Style where values are aligned

* E241 Multiple spaces after ',' - disabled because we often use 'C' Style where values are aligned

* E251 Unexpected spaces around keyword / parameter equals - disabled because it's broken so often it seems like a standard

* E261 At least two spaces before inline comment - disabled because it's broken so often it seems like a standard

* E266 Too many leading '#' for block comment - sometimes used as "section" separator

* E501 Line too long - disabled because it's broken so often it seems like a standard

* E701 Multiple statements on one line (colon) - broken only in convert.py when defining abstract methods (we can use# noqa instead)

* E704 Multiple statements on one line - broken only in convert.py when defining abstract methods (we can use# noqa instead)

main : Add ChatML functionality to main example (ggerganov#4046)

Co-authored-by: Sebastian Cramond <[email protected]>

readme : update ROCm Windows instructions (ggerganov#4122)

* Update README.md

* Update README.md

Co-authored-by: Jared Van Bortel <[email protected]>

---------

Co-authored-by: Jared Van Bortel <[email protected]>

finetune - update readme to mention llama support only (ggerganov#4148)

stablelm : simplify + speedup generation (ggerganov#4153)

docs : add llama-star arch idea

examples : fix typo in parallel example doc comment (ggerganov#4181)

Signed-off-by: Daniel Bevenius <[email protected]>

readme : update hot topics

llama : KV cache view API + better KV cache management (ggerganov#4170)

* llama : keep track of used KV cells + better KV cache management

* llama : zero KV cache used upon clear

ggml-ci

* llama : allow exporting a view of the KV cache (ggerganov#4180)

* Allow exporting a view of the KV cache

* Allow dumping the sequences per cell in common

* Track max contiguous cells value and position as well

* Fix max contiguous empty cells index calculation

Make dump functions deal with lengths or sequences counts > 10 better

* Fix off by one error in dump_kv_cache_view

* Add doc comments for KV cache view functions

Eliminate cell sequence struct; use llama_seq_id directly

Minor cleanups

* common : add -dkvc arg for enabling kv cache dumps

---------

Co-authored-by: Kerfuffle <[email protected]>

Fix incorrect format strings and uninitialized variables. (ggerganov#4133)

* Fix incorrect format strings and uninitialized variables.

* Address comments

* Add the missing include statement

readme : use PATH for Windows ROCm (ggerganov#4195)

* Update README.md to use PATH for Windows ROCm

* Update README.md

* Update README.md

main.swift : fix eos checking (ggerganov#4197)

llama_token_eos(const struct llama_model *) is currently getting struct llama_context type variable context as a parameter.

convert : fix tensors using grad in some models (ggerganov#4173)

ggml-cuda : support stablelm rope (ggerganov#4156)

* ggml-cuda : support stablelm rope

* remove unused freq_base kernel parameter

* add n_dims parameter to llm_build_k_shift, default to n_rot via overload

* llama : fix llm_build_k_shift args

---------

Co-authored-by: Georgi Gerganov <[email protected]>

llama : set metal log callback correctly (ggerganov#4204)

server : OAI API compatibility (ggerganov#4198)

* Add openai-compatible POST /v1/chat/completions API endpoint to server example

* fix code style

* Update server README.md

* Improve server README.md

* Fix server.cpp code style according to review

* server : some style changes

* server : indentation

* server : enable special tokens during tokenization by default

* server : minor code style

* server : change random string generator

* straightforward /v1/models endpoint

---------

Co-authored-by: kir-gadjello <[email protected]>
Co-authored-by: Tobi Lütke <[email protected]>

readme : update hot topics

Update docs for yarn_ext_factor <0.0 as unspecified instead of NaN (ggerganov#4189)

llama : grammar `reserve` space in `decode_utf8` (ggerganov#4210)

* reserve space for codepoints

* improvement for the appended 0

scripts : Use mmap in torch load (ggerganov#4202)

* Use mmap in torch load, prefer .bin files when loading

* Revert .bin > .safetensors preference

metal : fix yarn (ggerganov#4220)

get the correct n_orig_ctx in metal

lookahead : add example for lookahead decoding (ggerganov#4207)

* lookahead : init

* lookahead : generate and store n-grams

* lookahead : use loop instead recursion to generate n-grams

* lookahead : initial working implementation

* lookahead : filter repeating n-grams

* lookahead : use deterministic init

* lookahead : add to Makefile

* lookahead : fix a bug in the seq_id of the lookahead tokens

* lookahead : add comments

---------

Co-authored-by: slaren <[email protected]>

readme : update hot topics

lookahead : support `-n -1` infinite generation

ggml : fix -Warray-bounds warning with gcc (ggerganov#4231)

examples : iOS example with swift ui (ggerganov#4159)

* copy to llama.cpp as subdir

* attempt enabling metal, fails

* ggml metal compiles!

* Update README.md

* initial conversion to new format, utf8 errors?

* bug fixes, but now has an invalid memory access :(

* added O3, now has insufficient memory access

* begin sync with master

* update to match latest code, new errors

* fixed it!

* fix for loop conditionals, increase result size

* fix current workflow errors

* attempt a llama.swiftui workflow

* Update .github/workflows/build.yml

Co-authored-by: Georgi Gerganov <[email protected]>

---------

Co-authored-by: Georgi Gerganov <[email protected]>

readme : add Amica to UI list (ggerganov#4230)

cmake : fix issue with version info not getting baked into LlamaConfig.cmake (ggerganov#3970)

* Split CPP generation from build-info query

* Remove blank lines

* Add BUILD_SHARED_LIBS option

ggml : re-enable BLAS for CPU when src0 != F32 + remove redundant full offload checks in llama.cpp (ggerganov#4240)

* ggml : use blas even if src0 is not F32

* llama : use n_threads_batch only when n_tokens >= 32

ggml-ci

* llama : revert n_threads_batch logic

ggml-ci

ggml : restore abort() in GGML_ASSERT (ggerganov#4242)

readme : add FreeChat (ggerganov#4248)

examples : add readme files

py : fix oai proxy (ggerganov#3972)

* fix oai proxy

fix generation not stoped while bot stop talking in chat mode

fix possible `slot_id` not exist

response for cors (and pre flight)

* oai proxy: workaround for some client (such as Chatbox)

* use stop as separator to replace hardcoded `\n`

llama : fix typical sampling (ggerganov#4261)

Typical sampling was broken because after copying new_candidates into canditates, the "sorted" bool is left at "true", but the new data is no longer sorted according to probability. Patch to set "sorted" to false.

Test: Generating with temp=0.0001 (approx. argmax)  should generate the same sequence at typical>=1.0 and typical=0.9999 (approx. disabled, but enters the typical sampling codepath).

convert.py : fix llama/llama2 conversion due to vocab_size=-1 (ggerganov#4258)

llama : fix alignment of general.name in print meta (ggerganov#4254)

* llama: fix alignment of general.name in print meta

This commit fixes the alignment of the general.name field in the
llm_load_print_meta function.

Currently the output looks like this:
```console
llm_load_print_meta: model ftype      = mostly Q4_0
llm_load_print_meta: model params     = 13.02 B
llm_load_print_meta: model size       = 6.86 GiB (4.53 BPW)
llm_load_print_meta: general.name   = LLaMA v2
```
And with this commit it looks like this:
```console
llm_load_print_meta: model ftype      = mostly Q4_0
llm_load_print_meta: model params     = 13.02 B
llm_load_print_meta: model size       = 6.86 GiB (4.53 BPW)
llm_load_print_meta: general.name     = LLaMA v2
```

Signed-off-by: Daniel Bevenius <[email protected]>

* llama: fix alignment of special tokens

Signed-off-by: Daniel Bevenius <[email protected]>

---------

Signed-off-by: Daniel Bevenius <[email protected]>

readme : fix typo (ggerganov#4253)

llama.cpp uses GitHub Actions, not Gitlab Actions.

cmake : fix the metal file foder path (ggerganov#4217)

batched.swift : update README.md (ggerganov#4214)

docs: update how to run

docker : add finetune option (ggerganov#4211)

readme : fix (ggerganov#4135)

* fix: readme

* chore: resolve comments

* chore: resolve comments

main : pass LOG_TEE callback to llama.cpp log (ggerganov#4033)

* main : Call llama_log_set to use LOG_TEE

* tabs to spaces

llava : ShareGPT4V compatibility (vision encoder only loading) (ggerganov#4172)

* ShareGPT4 compatibility (vision encoder only loading)

Load only a CLIP vision encoder (as supplied by ShareGPT finetunes)
Corrects the argument parsing for --img_mean and --img_std (which were previously not parsed but attempted to access)
Defines defaults for img_mean and img_std which are equal to the llava 1.5 CLIP encoder, so you do not have to provide them

* Update convert-image-encoder-to-gguf.py

build : fix build info generation and cleanup Makefile (ggerganov#3920)

* cmake : fix joining of REAL_GIT_DIR

* fix includes with help from include-what-you-use

* make : remove unneeded deps and add test-rope target

* fix C includes in C++ source files

* Revert "fix includes with help from include-what-you-use"

This reverts commit 635e9fa.

make : fix Apple clang determination bug (ggerganov#4272)

Co-authored-by: Will Findley <[email protected]>

server : add single-client multi-prompt support (ggerganov#4232)

* * add multiprompt support

* * cleanup

* * more cleanup

* * remove atomicity of id_gen, and change lock_guard to unique_lock on completion requests

* * remove all references to mutex_multitasks

* Update examples/server/server.cpp

Co-authored-by: Jared Van Bortel <[email protected]>

* Update examples/server/server.cpp

Co-authored-by: Jared Van Bortel <[email protected]>

* Update examples/server/server.cpp

Co-authored-by: Jared Van Bortel <[email protected]>

* Update examples/server/server.cpp

Co-authored-by: Jared Van Bortel <[email protected]>

* * change to set

---------

Co-authored-by: Jared Van Bortel <[email protected]>

server : add --log-disable to disable logging to file (ggerganov#4260)

* * add --log-disable to disable logging to file in the server example

* * typo fix

ggml : add ggml_soft_max_ext (ggerganov#4256)

* metal : implement soft_max_ext

* cuda : implement soft_max_ext

* ggml : implement soft_max_ext (CPU)

* batched-bench : print threads

ggml-ci

* metal : simplify soft_max encoding

ggml-ci

* cuda : use 512 threads for soft_max instead of 32

* ggml : update soft max cpu

* cuda : do warp-based block reduce

* cuda : increase max block size to 1024

* cuda : fix warp reduction initialization of shared mem

* metal : warp-based reduction for soft max kernel

* metal : warp-based reduce for rms_norm

* metal : simplify soft max kernel

ggml-ci

* alloc : fix build with debug

py : add requirements file for convert-hf-to-gguf.py (ggerganov#4277)

This commit adds a requirements file for the convert-hf-to-gguf.py
script, and also add the torch and transformers packages to it.

The motivation for this is that currently running convert-hf-to-gguf.py
will produce the following error:
```console
$ python3 -m venv venv
$ source venv/bin/activate
(venv) $ pip install -r requirements.txt
Collecting numpy==1.24.4
Collecting sentencepiece==0.1.98
Collecting gguf>=0.1.0
Installing collected packages: sentencepiece, numpy, gguf
Successfully installed gguf-0.5.1 numpy-1.24.4 sentencepiece-0.1.98

(venv) $ python convert-hf-to-gguf.py --help
Traceback (most recent call last):
  File "llama.cpp/convert-hf-to-gguf.py", line 16, in <module>
    import torch
ModuleNotFoundError: No module named 'torch'
```
With this commit, and using requirements-hf-to-gguf.txt instead of
requirements.txt, the script can be run and shows the help output.

Signed-off-by: Daniel Bevenius <[email protected]>

llama : fix integer overflow during quantization (ggerganov#4284)

happens with multi-threaded quantization of Qwen-72B

ggml-ci

llama : add Qwen support (ggerganov#4281)

* enable qwen to llama.cpp

* llama : do not GPU split bias tensors

---------

Co-authored-by: Georgi Gerganov <[email protected]>

llama : support attention bias on LLaMA architecture (ggerganov#4283)

* Support attention_bias on LLaMA architecture

QKVO bias, should fix InternLM (ggerganov#3133) and works for LLaMAfied Qwen models (ggerganov#3743 (comment)).

* check existence of qkvo bias while loading llama models

Tested on LLaMA2, CUDA and CPU.

* Update llama.cpp

build : enable libstdc++ assertions for debug builds (ggerganov#4275)

swift : fix token_to_piece implementation (ggerganov#4278)

* Fix token_to_piece implementation in Swift

* Fix errors

llama : support optional tensors (ggerganov#4283)

llama : avoid using "optional" keyword (ggerganov#4283)

llama : pad KV cache size (ggerganov#4280)

* llama : pad KV cache size to 32

* metal : try to improve batched decoding

py : add grammar to oai like api (ggerganov#4294)

server : fix OpenAI API `stop` field to be optional (ggerganov#4299)

(cherry picked from commit Mozilla-Ocho/llamafile@e8c92bc)

ggml : fix soft max out-of-bounds access (ggerganov#4307)

ggml-ci

ggml : reuse ggml_get_n_tasks() in ggml_graph_plan() (ggerganov#4308)

* ggml : fix soft max out-of-bounds access

ggml-ci

* ggml : reuse ggml_get_n_tasks() in ggml_graph_plan()

ggml-ci

grammar-parser : fix typo (ggerganov#4318)

preceeding -> preceding

swift : fix prompt tokenization logic (ggerganov#4321)

swift : fix concatenation method to avoid invalid UTF8 stringfication (ggerganov#4325)

simple : update error message for KV cache check (ggerganov#4324)

This commit updates the error message that is printed when the
KV cache is not big enough to hold all the prompt and generated
tokens. Specifically it removes the reference to n_parallel and
replaces it with n_len.

Signed-off-by: Daniel Bevenius <[email protected]>

swift : revert compiler checks for swift package (ggerganov#4332)

sampling : custom samplers order (ggerganov#4285)

* Samplers sequence order w parameter

* Cleaned commented code

* Fixed formatting

* Rewrote with unordered_map

* Revert and rewrite, too many problems and safeguards would be needed

* Fixed code style

* Code style fixes according to review

* More readable samplers input string, fixed help

* Style fix in sampler_queue

* Formatting fixes

* Fixing whitespaces

llama : allow overriding GGUF metadata when loading model (ggerganov#4092)

* feat: Allow overriding GGUF metadata when loading model

* Fix the one time GCC is stricter than clang about something

* Step1

* Refactor... basically everything!

* Nuke obsolete GetArrayLen struct

* simplify std::string specialization

* Various cleanups

Add informational output when overrides are applied

Warn user when an override with the wrong type is specified

* Fix broken logic for parsing bool KV overrides
Fix issue where overrides didn't apply when key missing in GGUF metadata
Resolve merge changes

* llama : rearrange model params

* Update new GET_KEY call

Add note that metadata KV overrides aren't reflected in initial metadata KV info dump

---------

Co-authored-by: cebtenzzre <[email protected]>
Co-authored-by: Georgi Gerganov <[email protected]>

grammar : pre-computed pieces + reserve mem + less string copies (ggerganov#4330)

* reserve space for codepoints

* improvement for the appended 0

* used precomputed token text for grammar sample

* reserve canidates_decoded

* reserve canidates_grammar

* remove candidates_decoded

* Revert "remove candidates_decoded"

This reverts commit 3773328.

* changed decode_utf8 to take src by ref

speculative : support `--color` (ggerganov#4343)

* speculative: add some colors

* minor : add braces

---------

Co-authored-by: Georgi Gerganov <[email protected]>

common : fix compile warning

server : recognize cache_prompt parameter in OAI API (ggerganov#4347)

train : fix ggerganov#4227 (double free in examples/train-text-from-scratch/train-text-from-scratch.cpp) (ggerganov#4351)

On commit b1108 (44c117f) xaedes added

    ggml_allocr * alloc = NULL;

    ... (many lines in between)

    if (alloc) {
        ggml_allocr_free(alloc);
    }

Which is correct, but it's easy to lose context after many lines in between.

On commit b1287 (0e76a899) xaedes made a big change. From here on, alloc is freed eagerly.

    alloc = ggml_allocr_new(...)
    ... (short lines of code)
    ggml_allocr_free(alloc)

This happens a few times, but alloc is never set to NULL, and many lines below,
we still have

    if (alloc) {
        ggml_allocr_free(alloc);
    }

which causes a double-free.

llama : per-layer KV cache + quantum K cache (ggerganov#4309)

* per-layer KV

* remove unnecessary copies

* less code duplication, offload k and v separately

* llama : offload KV cache per-layer

* llama : offload K shift tensors

* llama : offload for rest of the model arches

* llama : enable offload debug temporarily

* llama : keep the KV related layers on the device

* llama : remove mirrors, perform Device -> Host when partial offload

* common : add command-line arg to disable KV cache offloading

* llama : update session save/load

* llama : support quantum K cache (ggerganov#4312)

* llama : support quantum K cache (wip)

* metal : add F32 -> Q8_0 copy kernel

* cuda : add F32 -> Q8_0 copy kernel

ggml-ci

* cuda : use mmv kernel for quantum cache ops

* llama : pass KV cache type through API

* llama : fix build

ggml-ci

* metal : add F32 -> Q4_0 copy kernel

* metal : add F32 -> Q4_1 copy kernel

* cuda : wip

* cuda : add F32 -> Q4_0 and F32 -> Q4_1 copy kernels

* llama-bench : support type_k/type_v

* metal : use mm kernel only for quantum KV cache

* cuda : add comment

* llama : remove memory_f16 and kv_f16 flags

---------

Co-authored-by: slaren <[email protected]>

* readme : add API change notice

---------

Co-authored-by: slaren <[email protected]>

sync : ggml (new ops, tests, backend, etc.) (ggerganov#4359)

* sync : ggml (part 1)

* sync : ggml (part 2, CUDA)

* sync : ggml (part 3, Metal)

* ggml : build fixes

ggml-ci

* cuda : restore lost changes

* cuda : restore lost changes (StableLM rope)

* cmake : enable separable compilation for CUDA

ggml-ci

* ggml-cuda : remove device side dequantize

* Revert "cmake : enable separable compilation for CUDA"

This reverts commit 09e35d0.

* cuda : remove assert for rope

* tests : add test-backend-ops

* ggml : fix bug in ggml_concat

* ggml : restore `ggml_get_n_tasks()` logic in `ggml_graph_plan()`

* ci : try to fix macOS

* ggml-backend : remove backend self-registration

* ci : disable Metal for macOS cmake build

ggml-ci

* metal : fix "supports family" call

* metal : fix assert

* metal : print resource path

ggml-ci

---------

Co-authored-by: slaren <[email protected]>

grammar : revert the replacement of llama_token_to_piece with id_to_token (ggerganov#4396)

Update README.md (ggerganov#4388)

Fix small typo.

ggml : increased GGML_MAX_PARAMS to allow finetuning of 70b models (ggerganov#4424)

server : fix local model name in server (ggerganov#4420)

llama : document logits_all deprecation (ggerganov#4418)

llama_context_params.logits_all is a parameter for controlling
llama_eval. This documents that logits_all should not be used with
llama_decode and llama_batch.

build : target Windows 8 for standard mingw-w64 (ggerganov#4405)

* build : target Windows 8 for standard mingw-w64

* make : fix missing console.o deps

This was causing a link error with `make all` on Windows.

english : use `typos` to fix comments and logs (ggerganov#4354)

server : tweak default sampling parameters (ggerganov#4367)

* Set a more typical Top P setting as the default

* Update temp max

llama : add Mixtral support (ggerganov#4406)

* convert : support Mixtral as LLAMA arch

* convert : fix n_ff typo

* llama : model loading

* ggml : sync latest ggml_mul_mat_id

* llama : update graph to support MoE

* llama : fix cur -> cur_expert

* llama : first working version

* llama : fix expert weighting in the FFN

* ggml : ggml_get_rows support 2D indexing [n_tokens, n_experts] (cpu only)

* ggml : add n_as argument to ggml_mul_mat_id

* ggml : fix ggml_get_rows to take into account ne02 / ne11

* metal : add more general support for ggml_get_rows + tests

* llama : add basic support for offloading moe with CUDA

* metal : add/mul/div use general kernel when src1 not cont

* metal : reduce the kernel launches for ggml_mul_mat_id

* ggml : get_rows : support non-contiguos tensors with gaps, generalize up to 3D

* ggml : update get_rows f16 and q

* cuda : support non-contiguous src1 in get_rows

* llama : offload missing ffn_moe_silu

* metal : fix ggml_get_rows to work with non-cont src1

* metal : add indirect mat-vec kernels for all quantization types

* llama : do not quantize expert gating tensors

* llama : add n_expert and n_expert_used to hparams + change quants

* test-backend-ops : add moe test

* cuda : fix get_rows when ncols is odd

* convert : determine n_ctx correctly

* metal : fix ggml_mul_mat_id for F32

* test-backend-ops : make experts more evenly probable (test_moe)

* test-backend-ops : cleanup, add moe test for batches

* test-backend-ops : add cpy from f32 -> all types test

* test-backend-ops : fix dequantize block offset

* llama : fix hard-coded number of experts

* test-backend-ops : simplify and disable slow tests to avoid CI timeout

* test-backend-ops : disable MOE test with thread sanitizer

* cuda : fix mul_mat_id with multi gpu

* convert : use 1e6 rope_freq_base for mixtral

* convert : fix style

* convert : support safetensors format

* gguf-py : bump version

* metal : add cpy f16 -> f32 kernel

* metal : fix binary ops for ne10 % 4 != 0

* test-backend-ops : add one more sum_rows test

* ggml : do not use BLAS with ggml_mul_mat_id

* convert-hf : support for mixtral-instruct (ggerganov#4428)

* convert : typo fix, add additional hyperparameters, use LLaMA arch for Mixtral-instruct

* convert : use sentencepiece tokenizer for Mixtral-instruct

* convert : make flake8 happy

* metal : fix soft_max kernels

ref: ggerganov/ggml@1914017

* metal : limit kernels to not use more than the allowed threads

---------

Co-authored-by: Georgi Gerganov <[email protected]>
Co-authored-by: Radek Pilar <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug-unconfirmed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@AndrewGodfrey @rvandernoort