Improvement: Running multiple test files at once

[Oscmage]

It would be lovely to run multiple test files at once similar to the github action.

Something like

fga model test --tests ./tests/*.fga.model

The github action got this feature, example:

      - name: Run Test
        uses: openfga/[email protected]
        with:
          test_files_pattern: "./tests/*.fga.yaml"

[7h3-3mp7y-m4n]

is it okay if I pick this?:)

[ewanharris]

@7h3-3mp7y-m4n I think it would be best for us to define this issue a little better before we ask someone to pick it up. But once we've done that it's all yours!

@Oscmage I'm curious what type of file you're passing to the command, is it a file that solely contains tests (as the command states it supports) or are you passing a store file like this.

My concern is that we may have overloaded the --tests option. It states Tests file Name. The file should have the OpenFGA tests in a valid YAML or JSON format but we generally direct folks to pass a store file (1, 2). By simply allowing multiple files to be passed to --tests, we would be supporting running tests against multiple different authorization models at once which I personally don't see the value in (outside of edge cases like sample-stores)

I believe that the value is more separating tests to help reduce file size, have individual "scenarios" per test file, or team ownership per test file, within a single authorization model project.

I'm wondering whether this issue could be broken down into a few parts that would support running multiple test files and help separate some boundaries between test files vs store files and local testing vs remote testing:

• Introduce a --store-file option that will accept a single store file only, it should parse similar to how --tests does currently
• Refactor --tests option to only support being passed multiple test file (i.e. a yaml or json file that doesn't contain anything other than tests)
  • We could then use this as our split between local vs remote testing that is unclear right now (Clarify that tests are run locally when specifying a model .fga.yaml + store-id #335), i.e. if you just use --tests we connect to an OpenFGA server to run them, f you pass --store-file we run locally.
• Support a test_files property in the store file similar to the model_file, tuple_file that exist already

[aaguiarz]

@ewanharris even if i agree that receiving multiple store.fga.yaml files with different models is not very useful, I also don't see why supporting it would be an issue. The reason OpenFGA users end create multiple fga.yaml files is usually because each of them tests a specific set of types (or modules), and in that case it makes sense to reuse the same model, the same tuples, and have different tests per file.

[Oscmage]

@ewanharris I would like to pass multiple files that looks like this https://github.com/openfga/cli/blob/main/example/model.fga.yaml . Or if you have a better way to achieve this I am happy to hear those!

What I am trying to solve is that we have multiple types in a model, we'd just like to split the tests for readability reasons and maintenance reasons. Basically all of the different files that I want to run have the same model_file but they might have different tuples setup for the different type that we want to test.

If I have 10 types it is natural to have 10 different test files to easily see the tests for each type. One huge test file makes it harder to read/maintain and reason about.

Happy to elaborate more if this is not clear enough. Again, there might be better ways to solve this problem, very open for that!

[ewanharris]

@Oscmage thanks for that feedback! I had imagine that was your usecase but just wanted to confirm.

I put together what the files could look like based on my comment this gist, instead of storing tests in a store file and running multiple store files you would instead list out your test files similar to how we support a model_file and tuple_file. You could then separate tests whatever fits bets, and if you're using modular models, you could colocate these test files with the various modules.

The main benefit I see of splitting into tests files over multiple store files is that you wouldn't need to copy around the model_file declaration every time you make a new test file (and also update the path if they are in different places). We'd continue to maintain a Store as a single top level entity in OpenFGA also which maybe would help with reasoning about things better.