setup/teardown at class and suite level

[bluebird75]

Today, luaunit supports setup/teardown to be executed before and after each test. Sometimes, it's convenient to also have per suite setup/teardown function and per class setup/teardown functions.

Proposal :

• names : setupSuite, teardownSuite, setupClass, teardownClass
• setupClass is always executed before running any test of the class
• teardownClass is always executed after running all tests of the class
• any failure or error in setupClass marks all tests as failed and does not execute them
• any failure or error in tearDownClass marks all tests as failed (only the successful tests, other are already failed)
  • note : it would be convenient to be able to report multiple error/failures for one test, to display a potential test failure, test teardown failure, test class teardown failure
• same principles for setupSuite and teardownSuite :
  • always executed, once per luaunit run invocation
  • failures put all tests into failures

Up for adoption...

[sskorupski]

Hi,
I'm wondering about taking in charge this enhancement proposal.

Need more information

Unfortunately I'm not used to use "class testing", I will be glad if anyone could share me a link to his sources showing how he uses it, so I will be able to understand the use-case.

Here is a schema about how I'm using Luaunit and how the test execution should be with the enhancement:
setupSuite,teardownSuite

As you can see, I use a single "test-suite.lua" file that have some requires to other test modules, using tables for grouping tests about the same "module under test"

Names

Watching the Luaunit source code, I can see that we can use test class, test function and test tables
so using setupClass and teardownClass for the naming should involve to use have also setupTable and teardownTable feature. But as far as the concept is the same I'm wondering about using a common beforeAll and afterAll methods that can be used in both class and table context. Keeping the idea that beforeAll is executed before all tests of the same test class/table and afterAll is executed after all tests of the same test class/table

Report multiple error/failures

note : it would be convenient to be able to report multiple error/failures for one test, to display a potential test failure, test teardown failure, test class teardown failure

Do you mean that if a test have few assertions, all the assertions should be evaluated and all failures are shown in the report ?
If you do, I disagree because of when an assertion fails, it's not usual to try to continue to evaluate the following instructions for the current test case because a failed assertion means that your system haven't the expected behaviour, so it's stopped. It's the way that other test frameworks work and so we should do.

[bluebird75]

Hi,

Thanks for volonteering.

You ask many questions, so let me take the time to answer.

There are no classes is Lua as you know, so what I actually call "class" is just a table grouping a set of tests. All the xUnit frameworks (Junit, Python unittest, cppunit ...) originate from the java unit test which was based on classes. That's why I am still using the concept of classes although it does not make total sense in the Lua world. The names setUp() and tearDown() also date back from this historical root. That's why I have kept them.

To be clearer about the context, let's imagine the following test code :

-- global suite functions
function setUpSuite()
end

function tearDownSuite()
end


-- class Alpha

TestClassAlpha = {}

function TestClassAlpha.setUpClass()
end

function TestClassAlpha.tearDownClass()
end

function TestClassAlpha.setUp()
end

function TestClassAlpha.tearDown()
end

function TestClassAlpha.testAlpha1() 
end

function TestClassAlpha.testAlpha2() 
end


-- class Beta

TestClassBeta = {}

function TestClassBeta.setUpClass()
end

function TestClassBeta.tearDownClass()
end

function TestClassBeta.setUp()
end

function TestClassBeta.tearDown()
end

function TestClassBeta.testBeta1() 
end

function TestClassBeta.testBeta2() 
end

The order of execution should be:

• setupSuite()
  • TestClassAlpha:setUpClass()
    • TestClassAlpha:setUp()
      • TestClassAlpha:testAlpha1()
    • TestClassAlpha:tearDown()
    • TestClassAlpha:setUp()
      • TestClassAlpha:testAlpha2()
    • TestClassAlpha:tearDown()
  • TestClassAlpha:tearDownClass()
  • TestClassBeta:setUpClass()
    • TestClassBeta:setUp()
      • TestClassBeta:testBeta1()
    • TestClassBeta:tearDown()
    • TestClassBeta:setUp()
      • TestClassBeta:testBeta2()
    • TestClassBeta:tearDown()
  • TestClassBeta:tearDownClass()
• tearDownSuite()

As you can see, there is no concept of setupTable which would be different from setupClass, it refers to the same topic.

For how to test this, it is quite simple : create side-effect in each function setup / teardown and test function, execute this whole subtest suite, and then verify that the side effects have been used in proper order. You can find such examples in test_luaunit.lua, line 3110, function TestLuaUnitExecution:testSetupAndTeardown()

I'll get back to you soon with regard to your other remarks. By the way, your link is not working for me...

[USSX-Hares]

@bluebird75 if I open a PR for that, would look at it, or have you stopped supporting the project?

Update #1

Looks these features are already in 3.4, but not documented.

Update #2

Naming convention in current implementation (3.4.1):

• setupClass
• teardownClass
• setupSuite
• teardownSuite

Noticed that if setupClass of first/only class fails, the LuaUnit enters the invalid state due to the NPE (call stack simplified):

luaunit.lua:2899: attempt to index a nil value (local 'node')
stack traceback:
> luaunit.lua:2899: in method 'updateStatus'
> luaunit.lua:3212: in method 'setupClass'
> luaunit.lua:2864: in method 'startClass'
> luaunit.lua:3073: in method 'execOneFunction'
> luaunit.lua:3252: in method 'internalRunSuiteByInstances'
[C]: in tests\testing\test-luaunit.lua:76: in main chunkion <luaunit.lua:3411>

Code sample:

    function TestClassAlpha.setupClass()
        error('!! TestClassAlpha.setupClass')
        print('> TestClassAlpha.setupClass')
    end
    function TestClassAlpha.teardownClass()
        print('> TestClassAlpha.teardownClass')
    end