Bug in v4.0.20 of oclif/core during testing with jest

[cmdcolin]

Describe the bug

It appears that running multiple tests no longer captures stdout properly in our projects test suite after upgrading v4.0.19 to v4.0.20 (released just today...bleeding edge :))

To Reproduce

clone repo, run tests https://github.com/cmdcolin/jest-oclif-test-bug-repro

Expected behavior

Tests pass

Screenshots

here is a session from my terminal showing running an individual test works, but running 'all tests' (both tests) fails. adding --runInBand does not fix it so it's not strictly a parallelism thing i think

 y test test/commands/hello/index.test.ts
yarn run v1.22.22
$ jest test/commands/hello/index.test.ts
 PASS  test/commands/hello/index.test.ts
  hello
    ✓ runs hello (219 ms)

Test Suites: 1 passed, 1 total
Tests:       1 passed, 1 total
Snapshots:   0 total
Time:        1.183 s, estimated 2 s
Ran all test suites matching /test\/commands\/hello\/index.test.ts/i.
Done in 1.56s.
~/mynewcli ❯❯❯ y test test/commands/hello/world.test.ts
yarn run v1.22.22
$ jest test/commands/hello/world.test.ts
 PASS  test/commands/hello/world.test.ts
  hello world
    ✓ runs hello world cmd (219 ms)

Test Suites: 1 passed, 1 total
Tests:       1 passed, 1 total
Snapshots:   0 total
Time:        1.166 s, estimated 2 s
Ran all test suites matching /test\/commands\/hello\/world.test.ts/i.
Done in 1.50s.
~/mynewcli ❯❯❯ y test
yarn run v1.22.22
$ jest
 FAIL  test/commands/hello/world.test.ts
  ● Console

    console.log
      hello world! (./src/commands/hello/world.ts)

      at Object.stdout (node_modules/@oclif/core/lib/ux/write.js:13:17)

  ● hello world › runs hello world cmd

    assert.strictEqual(received, expected)

    Expected value to strictly be equal to:
      undefined
    Received:
      ""

    Message:
      expected '' to include 'hello world!'

    Difference:

      Comparing two different types of values. Expected undefined but received string.

       6 |     const {stdout} = await runCommand('hello world')
       7 |     expect(stdout).to.contain('hello world!')
    >  8 |   })
         |     ^
       9 | })
      10 |

      at Object.<anonymous> (test/commands/hello/world.test.ts:8:39)

 FAIL  test/commands/hello/index.test.ts
  ● Console

    console.log
      hello friend from oclif! (./src/commands/hello/index.ts)

      at Object.stdout (node_modules/@oclif/core/lib/ux/write.js:13:17)

  ● hello › runs hello

    assert.strictEqual(received, expected)

    Expected value to strictly be equal to:
      undefined
    Received:
      ""

    Message:
      expected '' to include 'hello friend from oclif!'

    Difference:

      Comparing two different types of values. Expected undefined but received string.

       6 |     const {stdout} = await runCommand('hello friend --from oclif')
       7 |     expect(stdout).to.contain('hello friend from oclif!')
    >  8 |   })
         |     ^
       9 | })
      10 |

      at Object.<anonymous> (test/commands/hello/index.test.ts:8:39)

Test Suites: 2 failed, 2 total
Tests:       2 failed, 2 total
Snapshots:   0 total
Time:        1.534 s, estimated 2 s
Ran all test suites.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Environment (please complete the following information):

• ubuntu 24.04
• zsh 5.9
• using jest as the test runner

[mdonnalley]

@cmdcolin Thanks for the detailed writeup!

For context, we released a fix for this issue yesterday. The PR swapped process.stdout.write for console.log since using console.log doesn't seem to have the same back pressure issues that were reported with process.stdout.write

I was able to get your tests passing by setting verbose: true in the jest.config.js. According to the docs, verbose defaults to true if there's only one test to run but defaults to false if there are more than one. So that would explain why running a single test passed

I'm not familiar with jest, but my guess is that they're doing something to supress console.log when verbose is false and that's why @oclif/test can't capture what was printed - maybe?

I'm not sure that there's a whole lot we can do on the oclif side to avoid this - other than document that jest users should use verbose: true if they want to use @oclif/test

[cmdcolin]

interesting. the verbose:true workaround isn't too bad! thanks for finding

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[cmdcolin]

@mdonnalley the "verbose:true" workaround doesn't seem to work for jest anymore. i think <=4.0.25 worked but >=4.0.26 stopped working