Telescope panics when grep_string on a large project

[pierg75]

Description

nvim panics when opening telescope on a large project

Expected Behavior

It should not panic

Actual Behavior

nvim panics with the message:

PANIC: unprotected error in call to Lua API (not enough memory)

Details

Reproduce

1. move into a pretty big project (linux upstream source e.g.)
2. nvim -nu test.vim
3. open telescope with to grep_string
4. Do not search anything, just let the prompt there for a while

Environment

• nvim --version output:

Found existing alias for "nvim". You should use: "vi"
NVIM v0.5.0
Build type: RelWithDebInfo
LuaJIT 2.1.0-beta3
Compilation: /usr/bin/gcc -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -Wp,-U_FORTIFY_SOURCE -Wp,-D_FORTIFY_SOURCE=1 -DNVIM_TS_HAS_SET_MATCH_LIMIT -O2 -g -Og -g -Wall -Wextra -pedantic -Wno-unused-parameter -Wstrict-prototypes -std=gnu99 -Wshadow -Wconversion -Wmissing-prototypes -Wimplicit-fallthrough -Wvla -fstack-protector-strong -fno-common -fdiagnostics-color=auto -DINCLUDE_GENERATED_DECLARATIONS -D_GNU_SOURCE -DNVIM_MSGPACK_HAS_FLOAT32 -DNVIM_UNIBI_HAS_VAR_FROM -DMIN_LOG_LEVEL=3 -I/builddir/build/BUILD/neovim-0.5.0/x86_64-redhat-linux-gnu/config -I/builddir/build/BUILD/neovim-0.5.0/src -I/usr/include -I/usr/include/lua-5.1 -I/builddir/build/BUILD/neovim-0.5.0/x86_64-redhat-linux-gnu/src/nvim/auto -I/builddir/build/BUILD/neovim-0.5.0/x86_64-redhat-linux-gnu/include
Compiled by mockbuild@koji

Features: +acl +iconv +tui
See ":help feature-compile"

   system vimrc file: "$VIM/sysinit.vim"
  fall-back for $VIM: "/usr/share/nvim"

Run :checkhealth for more info

• Operating system: Fedora 34
• Telescope commit: 69adcbc

Configuration

set nocompatible hidden laststatus=2

if !filereadable('/tmp/plug.vim')
  silent !curl --insecure -fLo /tmp/plug.vim
    \ https://raw.githubusercontent.com/junegunn/vim-plug/master/plug.vim
endif

source /tmp/plug.vim
call plug#begin('/tmp/plugged')
Plug 'nvim-lua/popup.nvim'
Plug 'nvim-lua/plenary.nvim'
Plug 'nvim-telescope/telescope.nvim'
call plug#end()

autocmd VimEnter * PlugClean! | PlugUpdate --sync | close
lua require('telescope').setup()

[dhruvinsh]

builtin.tags() is also affected as well.

I have large tag file (807 Mb), and for it (Running :Telescope tags will throw the same error:

PANIC: unprotected error in call to Lua API (not enough memory)

[tjdevries]

This might be better now with the async stuff? we parse things as we get the results back instead of one huge chunk -- so maybe we won't run out of memory.

I guess we could implement some max items stuff, I don't remember if we have that or not.

[tjdevries]

(for grep_string, tags probably still has the problem)

[pierg75]

I just tested after update:

[telescope] [WARN 21:50:01] ...ck/packer/start/telescope.nvim/lua/telescope/pickers.lua:424: Finder failed with msg: not enough memory -- INSERT --PANIC: unprotected error in call to Lua API (not enough memory)

I'm not using the latest nvim source, if it matters though.

[tjdevries]

how many results are you getting before it crashes?

[tjdevries]

(and what would your desired behavior be)

[pierg75]

So, today I packaged latest 0.6 and I still get:
[telescope] [WARN 11:15:12] ...ck/packer/start/telescope.nvim/lua/telescope/pickers.lua:424: Finder failed with msg: not enough memory
The search I did returned 579 / 35148.
After some time it simply crashes again (not enough memory).
As for the behaviour, well, not crashing would be good enough.

[tjdevries]

So for example, good to do something like "limit results to 10,000" and notify / alert the user that not all the results will be displayed?

[pierg75]

Yeah, that sounds good.
Out of curiosity, what is the technical limit to show/store all the results? Are the results consuming lot of memory?

[GaetanLepage]

Hello,
I have this error when I do live_grep on my not-so-big project.
I have rg installed, and, from the terminal rg produces the result instantly.
However, in telescope, either it crashes (with the 'not enough memory' error) or it takes several seconds with one CPU core at 100%.

I tried live_grep on much bigger project (the cpython repo for instance) and it was really fast.
What would cause telescope live_grep to have such a hard time ? (probably my fault).

Here is my telescope config:

--------------
-- Bindings --
--------------
local nmap = require 'utils'.nmap

-- Find files using Telescope command-line sugar.
nmap('<leader>ff', '<cmd>Telescope find_files<CR>')
nmap('<leader>fg', '<cmd>Telescope live_grep<CR>')
nmap('<leader>fb', '<cmd>Telescope bufferts<CR>')
nmap('<leader>fh', '<cmd>Telescope help_tags<CR>')
nmap('<leader>fd', '<cmd>lua require("telescope.builtin").lsp_workspace_diagnostics()<CR>')

-- FZF like bindings
nmap('<C-p>', '<cmd>Telescope git_files<CR>')
nmap('<leader>p', '<cmd>Telescope oldfiles<CR>')
nmap('<leader>a', '<cmd>Telescope live_grep<CR>')


-------------------
-- Configuration --
-------------------
require 'telescope'.setup {
    defaults = {
        vimgrep_arguments = {
	        'rg',
	        '--no-heading',
	        '--with-filename',
	        '--line-number',
	        '--column',
	        '--smart-case'
    	},
        prompt_prefix = "> ",
        selection_caret = "> ",
        entry_prefix = "  ",
        initial_mode = "insert",
        selection_strategy = "reset",
        sorting_strategy = "descending",
        layout_strategy = "horizontal",
        layout_config = {
	        horizontal = {mirror = false},
	        vertical = {mirror = false}
    	},
        file_sorter = require'telescope.sorters'.get_fuzzy_file,
        file_ignore_patterns = {
            '.git',
            '__pycache__'
        },
        generic_sorter = require'telescope.sorters'.get_generic_fuzzy_sorter,
        path_display = {},
        winblend = 0,
        border = {},
        borderchars = {'─', '│', '─', '│', '╭', '╮', '╯', '╰'},
        color_devicons = true,
        use_less = true,
        set_env = {['COLORTERM'] = 'truecolor'}, -- default = nil,
        file_previewer = require'telescope.previewers'.vim_buffer_cat.new,
        grep_previewer = require'telescope.previewers'.vim_buffer_vimgrep.new,
        qflist_previewer = require'telescope.previewers'.vim_buffer_qflist.new,

        -- Developer configurations: Not meant for general override
        buffer_previewer_maker = require'telescope.previewers'.buffer_previewer_maker,
    }
}