fix(vitest): Define nested CAC options

[LorenzoBloedow]

I thought of different ways of addressing this but after realizing cac doesn't provide a native solution to transform option values with dot notation I thought the best way was to convert it to a boolean only if the string is either "true" or "false" and just ignore it if it's something different since Vitest doesn't seem to do runtime type checking.

I also thought about implementing this at the provider level but it'd mean the code would have to be duplicated for both v8 and istanbul.

It also doesn't seem like something the provider should be concerned with.

Regarding the tests, I wasn't sure how to proceed since currently there aren't any tests to differentiate between CLI and config file arguments so I'm not sure it's something the Vitest team wants.

I also think this is a bigger problem that includes other dot-notation based options like coverage.enabled and I'd like to hear your guys' opinions on this because it doesn't seem to be something cac can currently solve natively.

StackBlitz demonstrating that this fixes #3962 by using my own fork of this pull request: https://stackblitz.com/edit/vitest-dev-vitest-ecfukt

[stackblitz]

Run & review this pull request in StackBlitz Codeflow.

[netlify]

✅ Deploy Preview for fastidious-cascaron-4ded94 ready!

Name	Link
🔨 Latest commit	3c5acff
🔍 Latest deploy log	https://app.netlify.com/sites/fastidious-cascaron-4ded94/deploys/65c233b213f1d90008f3c841
😎 Deploy Preview	https://deploy-preview-3983--fastidious-cascaron-4ded94.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[AriPerkkio]

I wonder if there's a better way to do these using cac directly. Could it be configured to understand the object structures?

Regarding the tests, I wasn't sure how to proceed since currently there aren't any tests to differentiate between CLI and config file arguments so I'm not sure it's something the Vitest team wants.

We have some CLI flag tests at test/config but I'm not sure how this could be implemented there. We would need a way to print the resolved values of the test case and compare those depending on what CLI/config values were passed.

[LorenzoBloedow]

I wonder if there's a better way to do these using cac directly. Could it be configured to understand the object structures?
I'm not sure but I'll dig through the source later to find out, the docs seem to be lacking on that regard.

If I'm not mistaken, I think just doing coverage.thresholdAutoUpdate instead of coverage.thresholdAutoUpdate=true makes thresholdAutoUpdate a boolean, but I'm not sure and also [the docs currently suggest using true in dot notation].(https://vitest.dev/guide/coverage.html#other-options:~:text=enable-,coverage.enabled%3Dtrue,-in%20your%20configuration)
And even if it does, I'm also not sure if there's a way to pass false or even if it's a good idea to ignore the fact that people may not use the correct notation.

If cac can't do this natively I was thinking of forking it and adding that functionality so we can use it, what do you think?

[LorenzoBloedow]

@AriPerkkio Ok so I found out what the problem actually was.
mri (and by extension, cac) already does this conversion for us.

I haven't actually confirmed this but after digging through the source it seems cac will only parse options with dot notation properly if they're explicitly defined, then cac will use mri to parse the option which seems to do the string to boolean conversion for us.

Because of this, it seems this also affects any other property that uses dot notation and is not explicitly defined.
Should we turn this into a wider PR and define dot-notation based properties for all options?

[AriPerkkio]

That looks better and seems to work. Should we do this to all configuration options, @sheremet-va?

Here's test case that you can use to validate these. You might need to adjust some other files under test/config as well:

// vitest/test/config/test/options.test.ts
import { expect, test } from 'vitest'

import * as testUtils from '../../test-utils'

function runVitestCli(...cliArgs: string[]) {
  return testUtils.runVitestCli('run', 'fixtures/test/log-config.test.ts', ...cliArgs)
}

test('--coverage', async () => {
  const { stdout } = await runVitestCli('--coverage')

  expect(stdout).toMatch('coverage.enabled true')
})

test('--coverage.thresholdAutoUpdate', async () => {
  const { stdout } = await runVitestCli('--coverage.thresholdAutoUpdate=true')

  expect(stdout).toMatch('coverage.thresholdAutoUpdate true')
})

test('--coverage.thresholdAutoUpdate=false', async () => {
  const { stdout } = await runVitestCli('--coverage.thresholdAutoUpdate=false')

  expect(stdout).toMatch('coverage.thresholdAutoUpdate false')
})

// vitest/test/config/fixtures/test/log-output.test.ts
import { test } from 'vitest'
import type { UserConfig } from 'vitest/config'

/* eslint-disable no-console, unused-imports/no-unused-vars */

test('logs resolved configuration', async () => {
  // @ts-expect-error -- internal
  const { snapshotOptions, ...config }: UserConfig['test'] = globalThis.__vitest_worker__.config

  // Log options that are tested
  console.log('coverage.enabled', config.coverage?.enabled)
  console.log('coverage.thresholdAutoUpdate', config.coverage?.thresholdAutoUpdate)
})

[sheremet-va]

That looks better and seems to work. Should we do this to all configuration options, @sheremet-va?

But how does it look in the help menu? If it's displayed correctly, I'm fine with adding all options to CLI.

[AriPerkkio]

It's displayed correctly.

> vitest "--help"

Options:
  ...
  --coverage                             Enable coverage report 
  --coverage.thresholdAutoUpdate         Update threshold values: "lines", "functions", "branches" and "statements" to configuration file when current coverage is above the configured thresholds 
  ...

If listing all possible options makes --help too verbose, I think cli.help(callback) could be used to filter the message. https://github.com/cacjs/cac#clihelpcallback

[LorenzoBloedow]

If listing all possible options makes --help too verbose, I think cli.help(callback) could be used to filter the message. https://github.com/cacjs/cac#clihelpcallback

So should I filter dot-notation options?

[LorenzoBloedow]

Here's test case that you can use to validate these.

Shouldn't this be a concern for cac?
I think the most we should do is write snapshot tests for the shape of the CLI (is that already being done?), what do you think?

[AriPerkkio]

So should I filter dot-notation options?

Sure, we can then check if enabling them doesn't cause too verbose --help message.

Shouldn't this be a concern for cac?
I think the most we should do is write snapshot tests for the shape of the CLI

Feel free to implement test cases as you see best. The example shown above is printing Vitest's resolved configuration values, which means it will read CLI and configuration files and resolve final configuration object.

[LorenzoBloedow]

Sure, we can then check if enabling them doesn't cause too verbose --help message.

I think I did all the nested options, I changed the description on some of them where it makes sense to have a more compact description or a description with a link since CLIs don't have anchor tags.

I also thought about what you said regarding the verbose output so I went ahead and omitted the nested options by default but enabled them when the VERBOSE env variable is set.
I used an environment variable because I couldn't find a native way to add a verbose option to cac.
Maybe I could've added another command like verboseHelp but I thought this way made more sense because it adds less complexity, is more DRY and less prone to bugs, since I still use a callback from cac.

Let me know once you guys think this looks good so I can start writing the tests and updating the docs.

[AriPerkkio]

Filtering options from --help seems to work nice. When VERBOSE env is enabled, the output gets really verbose and reduces readability. I think it's OK to hide these new options completely from --help.

For the implementation I think we should structure the cac configuring so that options shown in --help are listed first, and the options excluded from --help are below.

cli
  .version(version)
  
  // Options shown in --help
  .option('-r, --root <path>', 'Root path')
  .option('-c, --config <path>', 'Path to config file')
  .option('--coverage', 'Enable coverage report')
  ...

  // Options hidden from --help. These are defined for type resolving.
  .option('--coverage.provider [name]', 'hide-from-help')
  .option('--coverage.enabled', 'hide-from-help')
  ...

Then in the .help(callback) it's easy to filter out by looking for 'hide-from-help'.

[LorenzoBloedow]

Filtering options from --help seems to work nice. When VERBOSE env is enabled, the output gets really verbose and reduces readability. I think it's OK to hide these new options completely from --help.

I disagree, when using the verbose option in any program, the user expects the output to be exactly that; verbose.
Since a "verbose level" system would be too complex, I placed a link to the docs where printing the whole explanation would otherwise become too verbose.

I think the combination of the links and disabling the verbose output by default, such that it won't ever get in the way of the user unless they explicitly choose to enable it, makes for very sensible defaults.

For the implementation I think we should structure the cac configuring so that options shown in --help are listed first, and the options excluded from --help are below.

I agree since it's probably easier to visually scan for the option you're looking for.

Then in the .help(callback) it's easy to filter out by looking for 'hide-from-help'.

This is a good idea if we do decide to go with the route of completely disabling the nested options from --help.

What do you think?

[AriPerkkio]

Is env.VERBOSE a convention across CLI programs? I haven't run into it before.

Instead of adding a verbose mode for --help it would be better to have option specific --help for options that support dot notation, e.g. --help --coverage to see all sub options of --coverage. I'm not sure if cac supports this.

[LorenzoBloedow]

Is env.VERBOSE a convention across CLI programs? I haven't run into it before.

Yeah I've never run into a CLI where VERBOSE would apply only to the --help command, using an environment variable might not be the best way.

Instead of adding a verbose mode for --help it would be better to have option specific --help for options that support dot notation, e.g. --help --coverage to see all sub options of --coverage. I'm not sure if cac supports this.

This is why I used an environment variable, from what I could gather on the docs and my own testing, cac searches for --help and prints the help output, it doesn't care what other options you put before and after it.

I think your idea of using --help --coverage would be the best way of dealing with this.
Unfortunately, cac doesn't seem to be actively maintained anymore, would a fork be appropriate? If so, I'm happy to start working on it!

[AriPerkkio]

Here's minimal example for filtering --help --<command>. I think this would be better than forking whole project.

import { cac } from "cac";

const cli = cac("repro");

cli
  .version("1.2.3")
  .option("-r, --root <path>", "Root path")
  .option("-u, --update", "Update snapshot")
  .option(
    "--coverage",
    "Enable coverage report. Use --help --coverage to see more options"
  )
  .option("--coverage.reporter [name]", "Choose coverage reporter")
  .help((info) => {
    const options = info.find((current) => current.title === "Options");
    if (typeof options !== "object") return info;

    const helpIndex = process.argv.findIndex((arg) => arg === "--help");
    const subCommand = process.argv[helpIndex + 1];

    // Filter out options that are not related to the sub command
    if (subCommand && subCommand.startsWith("--")) {
      options.body = options.body
        .split("\n")
        .filter((line) => line.trim().startsWith(subCommand))
        .join("\n");
    }
    // No sub-command, filter out options with dot-notation
    else {
      options.body = options.body
        .split("\n")
        .filter((line) => /--\S+\./.test(line) === false)
        .join("\n");
    }

    return info;
  });

cli.parse();

ari ~/repro  $ node cac.mjs --help 
repro/1.2.3

Usage:
  $ repro <command> [options]

Options:
  -v, --version               Display version number 
  -r, --root <path>           Root path 
  -u, --update                Update snapshot 
  --coverage                  Enable coverage report. Use --help --coverage to see more options 
  -h, --help                  Display this message 

ari ~/repro  $ node cac.mjs --help --coverage
repro/1.2.3

Usage:
  $ repro <command> [options]

Options:
  --coverage                  Enable coverage report. Use --help --coverage to see more options 
  --coverage.reporter [name]  Choose coverage reporter 

[LorenzoBloedow]

Here's minimal example for filtering --help --. I think this would be better than forking whole project.

Thanks for the great example! And while it's unfortunate we have to directly use process.argv (because it feels like something cac should be handling) I agree it's much better than forking the entire project.

I built on your example and added a few things (I already pushed them so you can check it out):

• Added support for providing multiple subcommands instead of just one
• Original help order is preserved even when using multiple subcommands in a random order
• You can use the --verbose option to display everything at once but it must be the only subcommand used
• Behavior that may be confusing to the user such as printing the whole help output when only invalid subcommands are found (imagine a typo) instead of printing nothing is shown with a warning explaining what happened

I may have gone a little overboard haha. Anyway, let me know your thoughts!

[sheremet-va]

• You can use the --verbose option to display everything at once but it must be the only subcommand used

I don't like using --verbose for this purpose. It's too ambiguous - vitest has a reporter with the same name for example. And jest also has this flag that shows more information about tests.

[LorenzoBloedow]

• You can use the --verbose option to display everything at once but it must be the only subcommand used

I don't like using --verbose for this purpose. It's too ambiguous - vitest has a reporter with the same name for example. And jest also has this flag that shows more information about tests.

What do you suggest? Maybe --show-all? Or are you saying to remove this option completely?

[sheremet-va]

Maybe --show-all?

Maybe --expand-help?

[LorenzoBloedow]

Maybe --expand-help?

Done!

Sorry for the delay and the force push, I messed up the git history and had to force push otherwise I would commit like 700 different changes!

Do we just need the tests now?

[sheremet-va]

Do we just need the tests now?

Ideally, yes. And mention this in docs.

I also see that browser tests are failing with this change - are you sure all CLI arguments are correct?

[LorenzoBloedow]

Ideally, yes. And mention this in docs.

Done.

I also see that browser tests are failing with this change - are you sure all CLI arguments are correct?

I forgot to put <name> in the --browser.name option, so that's fixed. But now one test failed and it seems like it might be a one-off issue, can we rerun the CI?

[sheremet-va]

Is it actually documented anywhere? I can't find how users should know about this command