segfault when using latest llama

[joprice]

I updated this lib to the latest llama, which I can build and run locally, but I'm getting a segfault when trying to run the simple example with my changes: master...joprice:llama-cpp-ocaml:update-llama-cpp. The master branch runs for me without issue, so I assume it's something with the bindings going out of sync that I didn't catch.

[joprice]

I added LLAMA_DEBUG=1 and found the spot of the segfault using lldb:

[joprice]

Looks like llama_batch's definition has changed to be a ** and a new field n_seq_id was added: ggerganov/llama.cpp@0e89203#diff-a2f09a47e379eeeb66a7398e3a1d11a391af75829d3e7a6dd7218e221b4fcaf3R140

[igarnier]

Thanks for your work! llama-cpp moves fast, it's hard to track all the changes. I'll try to look at your branch over the next few days, if you manage to get something working don't hesitate to submit a PR.

[joprice]

No problem. It would be great to be able to run llama from ocaml! I swapped over to the new multi-dimensional array for seq_id, and I'm no longer getting segfaults, but am hitting a pointer being freed was not allocated error 93a138b

* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
  * frame #0: 0x000000010013ff75 simple.exe`void std::__1::allocator<std::__1::__tree_node<int, void*>>::construct<int, int const&>(this=0x0000000100fd9010, __p=0x0000600000202f5c, __args=0x0000000000000000) at memory:1809:35
    frame #1: 0x000000010013ff40 simple.exe`void std::__1::allocator_traits<std::__1::allocator<std::__1::__tree_node<int, void*>>>::__construct<int, int const&>((null)=std::__1::true_type @ 0x00007ff7bfef1388, __a=0x0000000100fd9010, __p=0x0000600000202f5c, __args=0x0000000000000000) at memory:1687:21
    frame #2: 0x000000010013fc60 simple.exe`void std::__1::allocator_traits<std::__1::allocator<std::__1::__tree_node<int, void*>>>::construct<int, int const&>(__a=0x0000000100fd9010, __p=0x0000600000202f5c, __args=0x0000000000000000) at memory:1538:14
    frame #3: 0x000000010013f95f simple.exe`std::__1::unique_ptr<std::__1::__tree_node<int, void*>, std::__1::__tree_node_destructor<std::__1::allocator<std::__1::__tree_node<int, void*>>>> std::__1::__tree<int, std::__1::less<int>, std::__1::allocator<int>>::__construct_node<int const&>(this=0x0000000100fd9008, __args=0x0000000000000000) at __tree:2194:5
    frame #4: 0x000000010013f627 simple.exe`std::__1::pair<std::__1::__tree_iterator<int, std::__1::__tree_node<int, void*>*, long>, bool> std::__1::__tree<int, std::__1::less<int>, std::__1::allocator<int>>::__emplace_unique_key_args<int, int const&>(this=0x0000000100fd9008, __k=0x0000000000000000, __args=0x0000000000000000) at __tree:2139:29
    frame #5: 0x000000010013f532 simple.exe`std::__1::__tree<int, std::__1::less<int>, std::__1::allocator<int>>::__insert_unique(this=0x0000000100fd9008, __v=0x0000000000000000) at __tree:1269:16
    frame #6: 0x00000001000ae8db simple.exe`std::__1::set<int, std::__1::less<int>, std::__1::allocator<int>>::insert(this=0x0000000100fd9008 size=0, __v=0x0000000000000000) at set:659:25
    frame #7: 0x000000010011ac18 simple.exe`llama_kv_cache_find_slot(cache=0x000000011280e838, batch=0x00007ff7bfef1880) at llama.cpp:1517:48
    frame #8: 0x00000001000af7c4 simple.exe`llama_decode_internal(lctx=0x000000011280e800, batch=llama_batch @ 0x00007ff7bfef1880) at llama.cpp:5137:10
    frame #9: 0x00000001000b0558 simple.exe`llama_decode(ctx=0x000000011280e800, batch=llama_batch @ 0x00007ff7bfef1950) at llama.cpp:8953:21
    frame #10: 0x000000010009f02b simple.exe`llama_stub_38_llama_decode(x231=<unavailable>, x230=<unavailable>) at c_generated_functions.c:276:15 [opt]
    frame #11: 0x00000001002171af simple.exe`caml_c_call + 27

I'm not confident that I'm binding to the bigarray correctly. I also tried a nested pointer in the ctypes and then coercing it before building the big array, but that gave the same results.

Looks like this issue was fixed here by using a helper function in common.cpp ggerganov/llama.cpp#3803

[joprice]

Setting the type of seq_id to ptr (ptr Seq_id.repr) and then using the CArray api lets it succeed:

let seq_id = Ctypes.CArray.get seq_id 0 in
let seq_id = Ctypes.CArray.from_ptr seq_id 1 in
Ctypes.CArray.set seq_id 0 0l;

I'm not that familiar with Bigarray, so not sure if this nested pointer is possible to represent using it. It does give a better api, since now ctypes leaks into the use sites instead of the more generic and more user-friendly Bigarray.

[joprice]

I introduced a helper function to avoid leaking ctypes 7174abd. The simple example works now, but the repl fails with

Prompt [1]: hiAssertion failed: (sum > 0.0), fAssertion failed: (sum > 0.0), fAssertion failed: (sum > 0.0), function ggml_compute_forward_sofAssertion failed: (sum > 0.0), function ggml_compute_forward_soft_max_f32, file ggml.c, line 105unction ggml_compute_forward_sof84.
unction ggml_compute_forward_soft_max_f32, file ggml.c, line 105t_max_f32, file ggml.c, line 10584.
t_max_f32, file ggml.c, line 10584.

[joprice]

Looks like this last one is a missing call to n_seq_id.{i} <- Int32.of_int 1;. Updated