VM Nighly Tests are failing

[holgerd77]

VM Nighly Tests from here are currently failing on a daily basis, see e.g. this test run, respectively the raw logs since output is otherwise too large and truncated.

As far as I see all tests are passing with an ok state but then the test run ends with:

2023-01-03T03:24:07.3743646Z npm ERR! Lifecycle script `test:blockchain:allForks` failed with error: 
2023-01-03T03:24:07.3914471Z npm ERR! Error: command failed 
2023-01-03T03:24:07.3948982Z npm ERR!   in workspace: @ethereumjs/[email protected] 
2023-01-03T03:24:07.3990394Z npm ERR!   at location: /home/runner/work/ethereumjs-monorepo/ethereumjs-monorepo/packages/vm 
2023-01-03T03:24:07.5816710Z ##[error]Process completed with exit code 1.
2023-01-03T03:24:07.6219698Z Post job cleanup.

This might be something similar @acolytec3 fixed some time ago (can't completely remember when/how/where), Andrew, could you have a short look if this just might be the same return-value-or-something fix?

Otherwise: this should be generally tackled, so that we do not overlook any "real" errors.

[jochem-brouwer]

Nightly tests should fail on this test: #2406

[jochem-brouwer]

This is also in the nightly logs:

2023-01-03T01:49:47.5653940Z # file: ValidBlocks/bcExploitTest/ShanghaiLove.json test: ShanghaiLove_Homestead
2023-01-03T01:49:47.5782115Z ok 6758 correct genesis RLP
2023-01-03T01:49:48.1244879Z ok 6759 correct pre stateRoot
2023-01-03T01:50:34.0331747Z 
2023-01-03T01:50:34.0332751Z <--- Last few GCs --->
2023-01-03T01:50:34.0333245Z 
2023-01-03T01:50:34.0334174Z [2990:0x51cafe0]   244843 ms: Scavenge 2025.7 (2079.7) -> 2019.7 (2079.9) MB, 19.6 / 0.0 ms  (average mu = 0.645, current mu = 0.300) allocation failure 
2023-01-03T01:50:34.0335343Z [2990:0x51cafe0]   244911 ms: Scavenge 2026.6 (2079.9) -> 2020.4 (2080.7) MB, 31.6 / 0.0 ms  (average mu = 0.645, current mu = 0.300) allocation failure 
2023-01-03T01:50:34.0428217Z [2990:0x51cafe0]   245141 ms: Scavenge 2027.4 (2080.7) -> 2021.3 (2097.4) MB, 209.6 / 0.0 ms  (average mu = 0.645, current mu = 0.300) allocation failure 
2023-01-03T01:50:34.0428564Z 
2023-01-03T01:50:34.0428574Z 
2023-01-03T01:50:34.0428773Z <--- JS stacktrace --->
2023-01-03T01:50:34.0428950Z 
2023-01-03T01:50:34.0429415Z FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
2023-01-03T01:50:34.0429869Z  1: 0xb06730 node::Abort() [node]
2023-01-03T01:50:34.0430147Z  2: 0xa1b6d0  [node]
2023-01-03T01:50:34.0430622Z  3: 0xce1e60 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
2023-01-03T01:50:34.0431545Z  4: 0xce2207 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
2023-01-03T01:50:34.0431938Z  5: 0xe99875  [node]
2023-01-03T01:50:34.0432534Z  6: 0xea953d v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
2023-01-03T01:50:34.0433412Z  7: 0xeac23e v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
2023-01-03T01:50:34.0434221Z  8: 0xe6d77a v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
2023-01-03T01:50:34.0434889Z  9: 0x11e64e6 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
2023-01-03T01:50:34.0435300Z 10: 0x15da159  [node]
2023-01-03T01:51:36.3047474Z Aborted (core dumped)
2023-01-03T01:51:36.3070521Z npm ERR! Lifecycle script `tester` failed with error: 
2023-01-03T01:51:36.3074727Z npm ERR! Error: command failed 
2023-01-03T01:51:36.3077039Z npm ERR!   in workspace: @ethereumjs/[email protected] 
2023-01-03T01:51:36.3079579Z npm ERR!   at location: /home/runner/work/ethereumjs-monorepo/ethereumjs-monorepo/packages/vm 

[holgerd77]

Ah ok, wasn't aware that this is issue is still "in" and triggering the failure.

We should nevertheless act here in some way. From the perspective of nightly CI tests we just can't/shouldn't live with the situation that tests are failing daily, otherwise the tests won't be able to fulfill its purpose giving short-term notice on new bugs being introduced on old HFs.

So if we realistically judge that we cannot solve this issue for some more weeks to come, we should deactivate the test in the CI run. Would be even better of course if we can address so that things are just running normal again.

[holgerd77]

This has been fixed, will close.