[R-package] [ci] Test failure on CRAN's r-devel-windows-x86_64-gcc10-UCRT check

[jameslamb]

Description

One of the R package's unit tests is failing on CRAN's r-devel-windows-x86_64-gcc10-UCRT check.

── 1. Failure (test_lgb.Booster.R:469:5): Booster$eval() should work on a Datase
eval_in_mem not identical to eval_from_file.
Objects equal but not identical

full logs (click me)

using R Under development (unstable) (2021-10-11 r81035 ucrt)
using platform: x86_64-w64-mingw32 (64-bit)
using session charset: UTF-8
checking for file 'lightgbm/DESCRIPTION' ... OK
checking extension type ... Package
this is package 'lightgbm' version '3.3.0'
package encoding: UTF-8
checking package namespace information ... OK
checking package dependencies ... OK
checking if this is a source package ... OK
checking if there is a namespace ... OK
checking for executable files ... OK
checking for hidden files and directories ... OK
checking for portable file names ... OK
checking whether package 'lightgbm' can be installed ... OK
checking package directory ... OK
checking DESCRIPTION meta-information ... OK
checking top-level files ... OK
checking for left-over files ... OK
checking index information ... OK
checking package subdirectories ... OK
checking R files for non-ASCII characters ... OK
checking R files for syntax errors ... OK
checking whether the package can be loaded ... OK
checking whether the package can be loaded with stated dependencies ... OK
checking whether the package can be unloaded cleanly ... OK
checking whether the namespace can be loaded with stated dependencies ... OK
checking whether the namespace can be unloaded cleanly ... OK
checking loading without being on the library search path ... OK
checking dependencies in R code ... OK
checking S3 generic/method consistency ... OK
checking replacement functions ... OK
checking foreign function calls ... OK
checking R code for possible problems ... OK
checking Rd files ... OK
checking Rd metadata ... OK
checking Rd cross-references ... OK
checking for missing documentation entries ... OK
checking for code/documentation mismatches ... OK
checking Rd \usage sections ... OK
checking Rd contents ... OK
checking for unstated dependencies in examples ... OK
checking contents of 'data' directory ... OK
checking data for non-ASCII characters ... OK
checking data for ASCII and uncompressed saves ... OK
checking line endings in shell scripts ... OK
checking line endings in C/C++/Fortran sources/headers ... OK
checking line endings in Makefiles ... OK
checking compilation flags in Makevars ... OK
checking for GNU extensions in Makefiles ... OK
checking for portable use of $(BLAS_LIBS) and $(LAPACK_LIBS) ... OK
checking use of PKG_*FLAGS in Makefiles ... OK
checking compiled code ... OK
checking examples ... OK
checking for unstated dependencies in 'tests' ... OK
checking tests ... ERROR
  Running 'testthat.R'
Running the tests in 'tests/testthat.R' failed.
Complete output:
 REDACTED: 2000+ lines of test output
  ══ Warnings ════════════════════════════════════════════════════════════════════
REDACTED: 73 deprecation warnings
  ══ Failed ══════════════════════════════════════════════════════════════════════
  ── 1. Failure (test_lgb.Booster.R:469:5): Booster$eval() should work on a Dataset
  `eval_in_mem` not identical to `eval_from_file`.
  Objects equal but not identical
  
  ══ DONE ════════════════════════════════════════════════════════════════════════
  Error: Test failures
  Execution halted
checking PDF version of manual ... OK
DONE
Status: 1 ERROR

Reproducible example

Full logs: https://www.r-project.org/nosvn/R.check/r-devel-windows-x86_64-gcc10-UCRT/lightgbm-00check.html

Details on how CRAN runs this check: https://cran.r-project.org/web/checks/check_flavors.html#r-devel-windows-x86_64-gcc10-UCRT

Environment info

{lightgbm} 3.3.0 submitted to CRAN a few days ago (#4633 (comment)).

Additional Comments

Replicating this in CI

R Hub does not current support a build to replicate this CRAN check. They have a feature request for it at r-hub/rhub#482. (also checked this by running rhub::platforms() on the most recent release of {rhub}).

There is some background on the UCRT checks at https://developer.r-project.org/Blog/public/2021/03/12/windows/utf-8-toolchain-and-cran-package-checks/.

According to https://cran.r-project.org/web/checks/check_flavors.html#r-devel-windows-x86_64-gcc10-UCRT, the configuration of these checks is described in https://www.r-project.org/nosvn/winutf8/ucrt3/CRAN/checks/gcc10-UCRT/README.txt.

The specific test that's failing

The test that's failing is at

LightGBM/R-package/tests/testthat/test_lgb.Booster.R

Line 424 in d130bb1

test_that("Booster$eval() should work on a Dataset stored in a binary file", {

It was added in #4545 .

Based on the error message, I expect that the problem is coming from

LightGBM/R-package/tests/testthat/test_lgb.Booster.R

Line 469 in d130bb1

expect_identical(eval_in_mem, eval_from_file)

[jameslamb]

Given that the error says "Objects are equal but not identical", and that starting tomorrow I'll have limited availability for the next week and no access to a Windows environment, I'm tempted to just try "fixing" this by changing expect_identical() to expect_equal().

I expect that after some investigation we'll find that the different is something like "the encoding of the strings holding metric names changed slightly", or some other minor concern that isn't worth risking the package being removed from CRAN over.

[jameslamb]

It appears that CRAN re-ran this check on version 3.3.0 and {lightgbm} is now passing.

https://www.r-project.org/nosvn/R.check/r-devel-windows-x86_64-gcc10-UCRT/lightgbm-00check.html

https://cran.r-project.org/web/checks/check_results_lightgbm.html

This test should still be made more resilient to such issues to prevent future issues.

[StrikerRUS]

Closed via #4714.

[github-actions]

This issue has been automatically locked since there has not been any recent activity since it was closed.
To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues
including a reference to this.