test regression in 1.9.1

[anthraxx]

Hi, I've been trying to update the distro package to 1.9.1 and noticed a test regression which passed fine in 1.9.0
lua: 5.4.0

Running files in tests
Running tests/test-__vector.lua
Running tests/test-app.lua
Running tests/test-array2d.lua
Running tests/test-class.lua
Running tests/test-class2.lua
Running tests/test-class3.lua
Running tests/test-class4.lua
Running tests/test-compat.lua
Running tests/test-comprehension.lua
DONE
Running tests/test-config.lua
Running tests/test-data.lua
Running tests/test-data2.lua
Running tests/test-date.lua
lua5.4 -e 'package.path=[[lua/?.lua;lua/?/init.lua;]]..package.path'	lua5.4
Running tests/test-date2.lua
Running tests/test-dir.lua
Running tests/test-func.lua
Running tests/test-import_into.lua
Lua 5.4
Lua 5.4
Running tests/test-lapp.lua
Running tests/test-lexer.lua
Running tests/test-list.lua
Running tests/test-list2.lua
Running tests/test-map.lua
Running tests/test-multimap.lua
Running tests/test-orderedmap.lua
Running tests/test-path.lua
Running tests/test-pretty-number.lua
Running tests/test-pretty.lua
Running tests/test-seq.lua
Running tests/test-set.lua
Running tests/test-sip.lua
Running tests/test-strict.lua
Running tests/test-stringio.lua
Running tests/test-stringx.lua
Running tests/test-stringx2.lua
Running tests/test-tablex.lua
Running tests/test-tablex2.lua
permutations	5040
DONE
Running tests/test-tablex3.lua
Running tests/test-template.lua
template: success
Running tests/test-text.lua
Running tests/test-text2.lua
Running tests/test-types.lua
Running tests/test-url.lua
Running tests/test-utils.lua
warning: 'math.type' will not override existing symbol
Running tests/test-utils2.lua
Running tests/test-utils3.lua
Lua 5.4.0  Copyright (C) 1994-2020 Lua.org, PUC-Rio
sh: most-likely-nonexistent-command: command not found
Running tests/test-xml.lua
<hello><tag my_attribute='my_value'>dolly</tag></hello>
<hello><tag my_attribute='my_value'>dolly</tag></hello>
Running files in examples
Running examples/seesubst.lua
([see pl.bonzo.dog](pl.bonzo.html#dog))
remember about [see pl.bonzo](pl.bonzo.html#dog)

Running examples/sipscan.lua
SYNC%s*%[([+%-%d]%d*)%]%s*([+%-%d]%d*)%s+([+%-%d]%d*)	table: 0x5619d555a430	table: 0x5619d555a470
547	0
555	0
563	0
571	0
580	-1
587	0
Running examples/symbols.lua
Running examples/test-cmp.lua
true
true
Running examples/test-listcallbacks.lua
hello
yay
Running examples/test-pretty.lua
{
  "one",
  "two",
  "three",
  {
    1,
    2,
    3
  },
  ["&"] = true,
  _fred = {
    true,
    true
  },
  alpha = 1,
  beta = 2,
  gamma = 3,
  s = [[hello dolly
you're so fine
]],
  [0] = false
}
Running examples/test-symbols.lua
+ 	2 * x ^ 1 + 0
- 	2 * x
+ 	3 * 2
- 	6 * x
+ 	2 * x ^ 1 + 2 * 3
- 	2 * x + 6 * x ^ 2
+ 	2 * x ^ 1 + 2 * a
- 	6 * a + 2 * x
+ 	2 * a
- 	6 * a
+ 	1 * x + x * 1
- 	x ^ 2 * 3
Running examples/testclone.lua
---
true
/build/penlight/src/Penlight-1.9.1/examples/testclone.lua	/build/penlight/src/Penlight-1.9.1/copy/of/examples/examples/testclone.lua	1601072857	1601072847
lua5.4: examples/testclone.lua:38: assertion failed!
stack traceback:
	[C]: in function 'assert'
	examples/testclone.lua:38: in main chunk
	[C]: in ?
Running examples/testclone.lua failed with code 1

[alerque]

I can confirm this error cropped up between 1.9.0 and 1.9.1. Thanks for reporting it! However it is in an example, not in a true regression test. If I'm reading things rightly you are running lua run.lua as part of the Arch PKGBUILD's check() function; I think you need to run lua run.lua tests instead. I assume from the error above you already corrected the fact that (as of the 1.9.0 package in Arch) all split packages for different Lua versions were being tested with Lua 5.3.

Running just the regression suite and not the examples is probably sufficient for distro packaging purposes, and I hope that allows you to get Arch updated (yes it was me whining about Penlight not being updated for a year, and that was before I was a collaborator here).

However the issue report is still valid.

1. Our own CI test suite should probably have caught this and did not. We should run both the regression tests & in a separate job build the examples as well. I opened PR chore(ci): Test run examples as well as the regression suite #349 to address this.

2. The actual error bisects to 27ea5a2 and since examples/tescclone.lua uses pl.dir that makes sense. Perhaps @hishamhm you can review that change and spot what went wrong.

[anthraxx]

Yeah that has been corrected and it runs it for all lua variants now. For now it was on purpose, I thought running more (like examples) is better, but im fine just running the tests itself for now.

Is the behavior change in pl.dir that lead to breaking an example fine for penlight downstream consumption in a point release without expecting breakage?

[hishamhm]

@anthraxx The behavior change in #191 looks like an actual bug, I contributed #350 to fix it.