pending shouldn't fail tests

[Julian]

I don't really know anything about busted, so obviously let me know if I'm just misunderstanding the right way to use this but if I have a test with pending:

  it('expands mid-word', function()
    pending('norcalli/snippets.nvim#17')

    insert('(\\a<Tab>')
    assert.is.equal('(α', vim.fn.nvim_get_current_line())

    feed('ggdG')  -- FIXME: Start with clear buffers...
  end)

because I know it fails, and then I run it, I see

Starting...Scheduling: lua/tests/unicode_spec.lua

========================================
Testing:        @/Users/julian/Development/lean.nvim/lua/tests/unicode_spec.lua
Success ||      translations expands \-prefixed predefined substitutions on tab
Success ||      translations does not autoexpand
Pending ||      translations expands mid-word norcalli/snippets.nvim#17
Error detected while processing command line:
Fail    ||      translations expands mid-word
            .../julian/Development/lean.nvim/lua/tests/unicode_spec.lua:25: Expected objects to be equal.
            Passed in:
            (string) '(\a'
            Expected:
            (string) '(α'

            stack traceback:
                .../julian/Development/lean.nvim/lua/tests/unicode_spec.lua:25: in function <.../julian/Development/lean.nvim/lua/tests/unicode_spec.lua:21>


Success:        2
Failed :        1
Errors :        0
========================================
Tests Failed. Exit: 1

where the test I marked pending has failed the build. I see busted has a --suppress-pending, so if that's what's needed to get this not to fail maybe there just needs to be a way to enable that in plenary too, but why it's not the default seems weird to me. Maybe you'll enlighten me there :)

(Also it took a few minutes of looking for either "expected failure" or "todo" which are xUnit terms to finally find pending, but I see busted's docs don't use those terms anywhere)

[Conni2461]

plenary doesn't bundle busted anymore, we had problems with it. TJ basically wrote a simple busted with describe, it and pending and luassert for comparisons, etc...

pending actually works all you need to do is replace it with pending, the signature looks like this: pending(desc, func). You can find the whole simple busted spec file in tests/plenary/simple_busted_spec.lua.

About your "Start with clear buffer", would this work:

local bufnr = vim.api.nvim_create_buf(false, true) -- creates unlisted scratch buffer
vim.api.nvim_buf_call(bufnr, function()
  -- put everything you wanna do here
end)
vim.api.nvim_buf_delete(bufnr, { force = true })

You can obviously write a wrapper for that :)

Hope that answers your questions

[Julian]

Ahhh I see, it's not a statement you run in the body, you wrap the whole function with it. Thanks!

For the clear buffer thing, can I have plenary run that for every test then, or do I have to have that boilerplate everywhere?

[Julian]

I guess I can write a replacement wrapper for it that does that if I must... is that the way to do it?

[Conni2461]

You should just be able to do:

pending('expands mid-word', function()
  -- Waiting for 'norcalli/snippets.nvim#17'

  insert('(\\a<Tab>')
  assert.is.equal('(α', vim.fn.nvim_get_current_line())

  feed('ggdG')  -- FIXME: Start with clear buffers...
end)

Also i don't get why your buffers are not empty. How do you run your tests? Inside vim?
We just have this definied in a Makefile and that gives me empty buffers:

test:
	nvim --headless --noplugin -u scripts/minimal.vim -c "PlenaryBustedDirectory tests/plenary/ {minimal_init = 'tests/minimal_init.vim'}"

With minimal.vim:

set rtp+=.
set rtp+=../plenary.nvim/

runtime! plugin/plenary.vim

[Julian]

I have the same (I copied it from here), but state leaks from tests. E.g.:

⊙  cat lua/tests/leak_spec.lua                                                                                                                                        julian@Airm ●
local helpers = require('tests.helpers')
local feed, insert = helpers.feed, helpers.insert

describe('state leaks', function()
  it('inserts some text', function()
    assert.is.equal('', vim.fn.nvim_get_current_line())
    insert('foo bar baz')
  end)

  it('sees the text from the last test case', function()
    assert.is.equal('', vim.fn.nvim_get_current_line())
  end)
end)

~/Development/lean.nvim is a git repository on main
⊙  nvim --headless -c 'PlenaryBustedDirectory lua/tests/'                                                                                                             julian@Airm ●
Starting...Scheduling: lua/tests/unicode_spec.lua
Scheduling: lua/tests/leak_spec.lua

========================================
Testing:        @/Users/julian/Development/lean.nvim/lua/tests/leak_spec.lua
Success ||      state leaks inserts some text
Fail    ||      state leaks sees the text from the last test case
            /Users/julian/Development/lean.nvim/lua/tests/leak_spec.lua:11: Expected objects to be equal.
            Passed in:
            (string) 'foo bar baz'
            Expected:
            (string) ''

            stack traceback:
                /Users/julian/Development/lean.nvim/lua/tests/leak_spec.lua:11: in function </Users/julian/Development/lean.nvim/lua/tests/leak_spec.lua:10>


Success:        1
Failed :        1
Errors :        0
========================================
Tests Failed. Exit: 1

(Where locally I don't use the exact line you showed because I still don't know how y'all are doing that locally :), it's a bit annoying to set up, so locally I'm just running against my own vimrc, and in CI I run with the minimal .vimrc)

[Conni2461]

Oh yeah we don't clear the buffer after each it. I could ask @tjdevries if we should do that, if he doesn't take a look at this thread.

Edit: But that might cause problems with other tests. I am not sure
Edit 2: You should be able to clear the buffer with vim.api.nvim_buf_set_lines(0, 0, -1, false, {})

[Julian]

I think even beyond just clearing the buffer that it'd be nice if the default (or certainly a very easy tweak) got you stateless tests -- e.g. I wouldn't have expected setting buffer variables to persist across tests either.

But yeah if it's not the default if it's easy to make that happen without sprinkling boilerplate in every test maybe that'd be nice too.

EDIT: I don't know what the cultural norms of -spec-like test frameworks are though I guess -- maybe describe blocks are more expected to be stateless rather than individual it blocks? Or maybe neither?

[Conni2461]

I talked it over with tj and he suggested we should do something like this:

describe('thingy', function() 
  before_each(clear)
  -- And people can define there own before_each like this:
  before_each(function()
    -- also clear highlights, whatever
  end)
  -- and an inner describe can define additional before_each functions that will only run for that describe in addition to the already defined before_each
  it('should work', function() ... end) 
end)

I will do this in #50 but its already late for me and i am kinda tired so i will finish that tomorrow. :)

We can also talk about adding like an after_each

[Julian]

Cool! Sounds reasonable enough -- one "related" thing from experience (not with spec-like runners but with writing xUnit ones) if it helps, is that if you end up also adding after_each for teardown, it's slightly nice to also support an iterative way to do teardown too -- i.e. you want inside an it for the function in there to be able to say if foo then after_this(cleanup) done rather than needing to put all its teardown inside one after_each that runs the same for all tests and will require lots of guarding.

But for setup yeah that'd definitely help me here, thanks!