Can tests contain other tests, or provide input for other tests?

[cmeeren]

Not sure this makes sense, but I'll try.

I am using Expecto for running end-to-end tests against an API. One test checks for the ability to create an API key. Other tests need an API key, and exercise endpoints using this key. There is also a test for deleting an API key.

Currently I have well-known API keys in the DB and simply use those keys for the tests that need an API key, but I'd like to avoid this. I'd like (if it's possible and makes sense) for the "can create API key" test to supply the API key to the tests that need it, and for the "can delete API key" test to run at the end for cleaning up.

In other words, I have a need for a common setup/teardown for a list of tests, but where the setup/teardown themselves are tests.

(I'd rather avoid using per-test setup/teardown since that would create a unique API key for every single test and thus triple the amount of requests.)

Is this possible? Does it make sense? I see from the readme that there's such a wealth of ways of defining/structuring/combining tests that it makes my head spin.

[haf]

let myTests (keyId: string) =
  testList [ ... ]
let main argv =
  use key = setupKey ()
  runTestsWithArgs defaultConfig argv (myTests key.id)

[cmeeren]

Thanks, but is setupKey a test? Will it show in the test explorer and test results?

To be specific: I'd like to have a test "can create API key" and a test "can delete API key", and ideally, these are the only calls to the create/delete API key endpoints.

A workaround is, as it seems you are suggesting, to use setup/teardown but also have the "can create API key" and "can delete API key" as separate tests. As I understand it, this means that a total of two API keys will be created/deleted. (I can live with that, but I am wondering if it can be avoided.)

[haf]

There's really not anything "obvious" that solves your problem inside the test framework; but since you can program anything, you can choose your solution. My suggestion was that you put that logic outside of your running of the tests, so that setupKey sets up the key, and when the tests finish, the key gets deleted.

[cmeeren]

Thanks! That's what I wanted to know.

I'm using [<Tests>] for VS discovery/running so AFAIK I can't have stuff outside my tests like that, but I'm sure I can get some setup/teardown working anyway.

[haf]

@cmeeren Would it help to have an event system in Expecto? E.g. with "test starting", "test running", "test completed", "n/m tests completed", etc?

[cmeeren]

Well, it would be a powerful feature that would make it possible to create dependent tests (by setting mutable shared state), though I'm not in any way convinced it's a good idea to couple tests imperatively that way. (I just wanted to know if it was already supported in some manner.) Personally I'm having a hard enough time as it is wrapping my head around structuring tests in Expecto, setup/teardown/fixtures/parametrized etc. (and I'm currently the most F# proficient in our company, so I may still simply end up using another test framework to make it easier on everyone else.)

[cmeeren]

I have now understood testFixture for setup/teardown for each test in a list, but is it possible to use it for common setup/teardown for all tests in a list?

[haf]

Ok, I see what you're saying. So off the top of my head, perhaps something like:

let testListWith testListName disposableFactory (tests: ('thing -> Test) list) =
  testSequenced <| testList (sprintf "wrapper for %s" testListName) [
    let value = disposableFactory ()
    testList testListName (tests |> List.map (fun factory -> factory value))
    testCase (sprintf "%s disposing" testListName) <| fun () ->
      (value :> IDisposable).Dispose()
  ]

Usage:

let createAPIKey () =
  APIKey.create "deadbeef" |> API.Auth.authenticate

let apiTestCase name testMethod =
  fun apiKey ->
    testCaseAsync name (testMethod apiKey)

[<Tests>]
let tests =
  testListWith "API Key" createAPIKey [
    apiTestCase "can fetch list" <| fun apiKey -> async {
      let! listing = API.FooBars.list apiKey [ Params.PageSize 10 ]
      listing |> Expect.isNonEmpty "Should return at least one item"
    }
  ]

[cmeeren]

Thanks a lot! The flexibility of Expecto is nice.

I would also like the actual tests using the API key (between setup/teardown) to run in parallel. I tried to place them in a "child" test list inside the list passed to testSequenced and expected this inner list to run in parallel, but it seems it too is run in sequence. Is there any way to run a subset of testSequenced in parallel?

[joprice]

I was curious how to get this working for a setup function that requires Task, like a testcontainer or a local server, and got this variant working, where the setup function returns a task and another task to cleanup. Maybe someone will find it useful:

  let integrationTestFixtureList (name: string) (setup: unit -> Task<'b * (unit -> 'c)>) tests =
     testSequenced
      <| testList (sprintf "wrapper for %s" name) [
        let value = setup ()
        testList
          name
          (tests
           |> List.map (fun (name, factory) ->
             testCaseTask
               name
               (fun () -> task {
                 let! value, _ = value
                 do! factory value
               })
           ))
        testCaseTask (sprintf "%s disposing" name)
        <| fun () -> task {
          let! _, dispose = value
          do! dispose ()
        }
      ]