bug: Fail to get tokenizer for Cohere provider: "response too big for into_string"

[maxbrunet]

Describe the bug

The error happens shortly after startup:

thread '<unnamed>' panicked at crates/avante-tokenizers/src/lib.rs:74:73:
called `Result::unwrap()` on an `Err` value: Custom { kind: Other, error: "response too big for into_string" }
stack backtrace:
   0: _rust_begin_unwind
   1: core::panicking::panic_fmt
   2: core::result::unwrap_failed
   3: avante_tokenizers::from_pretrained
   4: mlua::state::Lua::create_function::{{closure}}
   5: mlua::state::util::callback_error_ext
   6: mlua::state::raw::RawLua::create_callback::call_callback
   7: <unknown>
   8: _lua_pcall 
   9: _nlua_schedule_event
  10: _state_handle_k_event
  11: _nv_event
  12: _normal_execute
  13: _state_enter
  14: _normal_enter
  15: _main
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

avante.nvim/crates/avante-tokenizers/src/lib.rs

Line 74 in e98fa46

let _ = std::fs::write(&cached_path, response.into_string().unwrap());

This happens since PR #765 (before the issue was #499)

To reproduce

Open any non-empty file, the error should occur after a few seconds after trying to get the tokenizer

Expected behavior

Avante is able to start with the Cohere provider

Installation method

Use lazy.nvim:

{
  "yetone/avante.nvim",
  event = "VeryLazy",
  lazy = false,
  version = false,
  build = "make",
  opts = {
    provider = "cohere",
    auto_suggestions_provider = "cohere",
  },
  dependencies = {
    "stevearc/dressing.nvim",
    "nvim-lua/plenary.nvim",
    "MunifTanjim/nui.nvim",
  },
}

CO_API_KEY and HF_TOKEN environment variables set.

Environment

Neovim v0.10.2
macOS 15.1

Repro

vim.env.LAZY_STDPATH = ".repro"
load(vim.fn.system("curl -s https://raw.githubusercontent.com/folke/lazy.nvim/main/bootstrap.lua"))()

require("lazy.minit").repro({
  spec = {
    {
      "yetone/avante.nvim",
      event = "VeryLazy",
      lazy = false,
      version = false,
      build = "make",
      opts = {
        provider = "cohere",
        auto_suggestions_provider = "cohere",
      },
      dependencies = {
        "stevearc/dressing.nvim",
        "nvim-lua/plenary.nvim",
        "MunifTanjim/nui.nvim",
      },
    }
  },
})