Silently ignoring syntax error in test file?

[krlmlr]

To replicate, run the "numbers" tests in r-lib/pillar@3611c62 via testthat::test_local(filter = "numbers"). Can't (easily) create a reprex here.

The syntax error is tibble::tibble::tibble, which is rejected by the parser in a clean session.

[gaborcsardi]

It is a bug in parallel testthat.

❯ Sys.setenv(TESTTHAT_PARALLEL = "FALSE")
❯ testthat::test_local(filter = "numbers")
✔ |  OK F W S | Context
⠏ |   0       | numbers                                                         
Error in parse(con, n = -1, srcfile = srcfile, encoding = "UTF-8") :
  test-numbers.R:17:19: unexpected '::'
16:     # FIXME: Use fixed digits
17:     tibble::tibble::

[billdenney]

I have a bug where a syntax error in a file causes testthat parallel to hang forever. The issue was a missing parenthesis, and there was no notification in the main process.

Is that the same as this?

[gaborcsardi]

@billdenney Possibly.

[billdenney]

@gaborcsardi , I just tried to make a reprex, and I got this behavior. So, I think that either my issue was the same or it's fixed.

[kamilzyla]

I have just been bitten by this. It seems that testthat::test_local() ignores all files with syntax errors when running in parallel. To reproduce, run the following in an empty directory:

usethis::create_package(".")
usethis::use_testthat(parallel = TRUE)
usethis::use_test("syntax")

# Whatever to create a syntax error
write("@", "tests/testthat/test-syntax.R", append = TRUE)

# Suceeds
testthat::test_local()

Tested with testthat 3.2.1.

[orichters]

I noticed the same. I think that is a major flaw. How to reproduce under testthat 3.2.3: Add a new file tests/testthat/test-new.R with the following content:

test_that("new", {
  expect_no_warning(a <- ()
})

The wrong file is just ignored and the tests pass, if I use these settings in the description:

Config/testthat/parallel: true
Config/testthat/edition: 3

If I switch to Config/testthat/parallel: false, it throws an error:

Error in `parse()`:
! test-new.R:2:27: unexpected ')'
1: test_that("new", {
2:   expect_no_warning(a <- ()
                             ^
Backtrace:
     ▆
  1. ├─devtools::test(show_report = TRUE)
  2. │ └─testthat::test_local(...)
  3. │   └─testthat::test_dir(...)
  4. │     └─testthat:::test_files(...)
  5. │       └─testthat:::test_files_serial(...)
  6. │         ├─testthat::with_reporter(...)
  7. │         │ └─base::tryCatch(...)
  8. │         │   └─base (local) tryCatchList(expr, classes, parentenv, handlers)
  9. │         │     └─base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]])
 10. │         │       └─base (local) doTryCatch(return(expr), name, parentenv, handler)
 11. │         └─base::lapply(...)
 12. │           └─testthat (local) FUN(X[[i]], ...)
 13. │             └─testthat::source_file(path, env = env(env), desc = desc, error_call = error_call)
 14. │               └─base::parse(con, n = -1, srcfile = srcfile, encoding = "UTF-8")
 15. └─base::.handleSimpleError(...)
 16.   └─h(simpleError(msg, call))
 17.     └─rlang::entrace(cnd)
 18.       └─rlang::cnd_signal(entraced)
Execution halted
make: *** [Makefile:23: test] Error 1

@hadley: as the most active developer, might I point you to this error which I think is quite severe. Thanks!