maintenance: failing tests #1357

s1fr0 · 2022-11-08T14:04:02Z

Problem

In the development of nwaku, we occasionally observe some CI/Jenkins test failures that are usually not triggered when tests are re-executed.

This issue aims to collect observed test failures in order to:

track context in which a failure happened;
discuss/speculate possible causes that triggered the failure;
make developers aware if a failure is new (and likely coming from recently changed code) or was previously known.

Everyone is encouraged to edit this issue and add new tests that fail during checks.
Once the causes of a test failure are identified and addressed, please remove the failing test from this post and comment to this issue by reporting it description, the list of identified causes and link the PR that fixed it.

Any feedback is welcome!

Test report template

Action type: CI/Jenkins
OS: linux/macOS
Test name/version: (v1/v2)
nwaku commit/tree:
Log excerpt:
Causes: (not identified/speculated/identified)
TimesSeen: (times the test was seen)
Temporary fix/workaround: (if any)

Failing tests

Action type: CI
OS: macOS
Test name/version: "Messages are bridged between Waku v1 and Waku v2" (v2)
nwaku commit/tree: https://github.com/status-im/nwaku/tree/7bc657b545c1b6eb54d5b70f8a926fb208aad939
Log excerpt:
TimeSeen: 4

2022-11-08T12:27:30.0160480Z     /Users/runner/work/nwaku/nwaku/tests/v2/test_waku_bridge.nim(152, 49): Check failed: v1Node.protocolState(Waku).queue.items.len == 1
2022-11-08T12:27:30.0161010Z     v1Node.protocolState(Waku).queue.items.len was 0
2022-11-08T12:27:30.0161840Z /Users/runner/work/nwaku/nwaku/vendor/nimbus-build-system/vendor/Nim/lib/system/fatal.nim(53) sysFatal
2022-11-08T12:27:30.0162530Z /Users/runner/work/nwaku/nwaku/vendor/nim-chronos/chronos/asyncfutures2.nim(391) futureContinue
2022-11-08T12:27:30.0162820Z 
2022-11-08T12:27:30.0163040Z     Unhandled exception: index out of bounds, the container is empty [IndexDefect]
2022-11-08T12:27:30.0163440Z   [FAILED] Messages are bridged between Waku v1 and Waku v2

Causes: not identified

Action type: CI
OS: macOS
Test name/version: "event subscription" (v2)
nwaku commit/tree: https://github.com/status-im/nwaku/tree/63137f3e2a9457584276823918d55065f7d8e3bf
Log excerpt:
TimeSeen: 1

DBG 2022-11-09 20:38:28.017+00:00 Address of the deployed membership contract:  tid=179666 file=test_waku_rln_relay_onchain.nim:73 contractAddress=0xd12eac7850902252129eb346dc3d5e0d279e02a4
DBG 2022-11-09 20:38:28.019+00:00 Account balance after the contract deployment:  tid=179666 file=test_waku_rln_relay_onchain.nim:76 balance=9999997207396538075888
DBG 2022-11-09 20:38:28.019+00:00 disconnected from                          tid=179666 file=test_waku_rln_relay_onchain.nim:79 ethClientAddress=ws://127.0.0.1:8540
DBG 2022-11-09 20:38:28.020+00:00 web3 connected to                          tid=179666 file=test_waku_rln_relay_onchain.nim:184 EthClient=ws://127.0.0.1:8540
DBG 2022-11-09 20:38:28.027+00:00 contract deployer account address          tid=179666 file=test_waku_rln_relay_onchain.nim:189 defaultAccount=0xcbe069e7c30e77b3e17f1929236bcd87e3b89f3b
DBG 2022-11-09 20:38:28.772+00:00 membership commitment key                  tid=179666 file=test_waku_rln_relay_onchain.nim:206 pk=14109699623625531147653880372921537588611513249833251812463573252554215275504
Error from subscription: key not found: data
DBG 2022-11-09 20:38:28.804+00:00 a member is registered                     tid=179666 file=test_waku_rln_relay_onchain.nim:226 tx=0x3fcdc383108516230c6867d0708f1f5a947427475e4947a1c183b287fc3a6298
2022-11-09T20:57:19.5714790Z Error: The operation was canceled.

Causes: not identified

Action type: CI
OS: macOS
Test name/version: "peer subscription should not be dropped if connection recovers before timeout elapses" (v2)
nwaku commit/tree: https://github.com/status-im/nwaku/tree/71c5bfda0e4bfbedd967ed23e2cb53899de10c11
Log excerpt:
TimeSeen: 4

2022-11-10T16:44:18.4130470Z     /Users/runner/work/nwaku/nwaku/tests/v2/test_waku_filter.nim(236, 18): Check failed: await pushHandlerFuture.withTimeout(1.seconds)
2022-11-10T16:44:18.4146180Z stack trace: (most recent call last)
2022-11-10T16:44:18.4146930Z /Users/runner/work/nwaku/nwaku/vendor/nimbus-build-system/vendor/Nim/lib/system/nimscript.nim(429, 18)
2022-11-10T16:44:18.4148050Z /Users/runner/work/nwaku/nwaku/waku.nimble(76, 8) test2Task
2022-11-10T16:44:18.4148690Z /Users/runner/work/nwaku/nwaku/waku.nimble(42, 8) test
2022-11-10T16:44:18.4149300Z /Users/runner/work/nwaku/nwaku/vendor/nimbus-build-system/vendor/Nim/lib/system/nimscript.nim(273, 7) exec
2022-11-10T16:44:18.4149980Z /Users/runner/work/nwaku/nwaku/vendor/nimbus-build-system/vendor/Nim/lib/system/nimscript.nim(273, 7) Error: unhandled exception: FAILED: build/all_tests_v2 [OSError]

Causes: not identified

Action type: CI
OS: macOS
Test name/version: "peer subscription should be dropped if connection fails for second time after the timeout has elapsed" (v2)
nwaku commit/tree: https://github.com/waku-org/nwaku/tree/b38bf152fc50e762a3db098a49eb05f253be76a7
Log excerpt:
TimeSeen: 1

2022-12-14T09:46:29.9735490Z     /Users/runner/work/nwaku/nwaku/tests/v2/test_waku_filter.nim(172, 18): Check failed: await pushHandlerFuture.withTimeout(1.seconds)
2022-12-14T09:46:29.9835300Z stack trace: (most recent call last)
2022-12-14T09:46:29.9850710Z /Users/runner/work/nwaku/nwaku/vendor/nimbus-build-system/vendor/Nim/lib/system/nimscript.nim(429, 18)
2022-12-14T09:46:29.9948520Z /Users/runner/work/nwaku/nwaku/waku.nimble(76, 8) test2Task
2022-12-14T09:46:29.9990240Z /Users/runner/work/nwaku/nwaku/waku.nimble(42, 8) test
2022-12-14T09:46:29.9995840Z /Users/runner/work/nwaku/nwaku/vendor/nimbus-build-system/vendor/Nim/lib/system/nimscript.nim(273, 7) exec
2022-12-14T09:46:30.0098480Z /Users/runner/work/nwaku/nwaku/vendor/nimbus-build-system/vendor/Nim/lib/system/nimscript.nim(273, 7) Error: unhandled exception: FAILED: build/all_tests_v2 [OSError]

Causes: not identified (root cause seems to be the same as previous failing test)

Action type: CI
OS: linux
Test name/version: "Admin API: connect to ad-hoc peers" (v2)
nwaku commit/tree: https://github.com/waku-org/nwaku/tree/b38bf152fc50e762a3db098a49eb05f253be76a7
Log excerpt:

2022-12-14T09:14:26.3263997Z /home/runner/work/nwaku/nwaku/vendor/nim-json-rpc/json_rpc/client.nim(467) main
2022-12-14T09:14:26.3264712Z /home/runner/work/nwaku/nwaku/vendor/nim-json-rpc/json_rpc/client.nim(457) NimMain
2022-12-14T09:14:26.3265115Z /home/runner/work/nwaku/nwaku/vendor/nim-json-rpc/json_rpc/client.nim(448) PreMain
2022-12-14T09:14:26.3265524Z /home/runner/work/nwaku/nwaku/vendor/nim-json-rpc/json_rpc/client.nim(116) PreMainInner
2022-12-14T09:14:26.3265946Z /home/runner/work/nwaku/nwaku/vendor/nim-unittest2/unittest2.nim(848) suitePayload
2022-12-14T09:14:26.3266387Z /home/runner/work/nwaku/nwaku/vendor/nim-testutils/testutils/unittests.nim(16) runTestX60gensym680
2022-12-14T09:14:26.3266818Z /home/runner/work/nwaku/nwaku/vendor/nim-testutils/testutils/unittests.nim(15) scenario
2022-12-14T09:14:26.3267262Z /home/runner/work/nwaku/nwaku/vendor/nim-chronos/chronos/asyncfutures2.nim(372) futureContinue
2022-12-14T09:14:26.3267597Z /home/runner/work/nwaku/nwaku/tests/v2/test_jsonrpc_waku.nim(375) scenario
2022-12-14T09:14:26.3267934Z /home/runner/work/nwaku/nwaku/waku/v2/node/jsonrpc/admin_api.nim(33) constructMultiaddrStr
2022-12-14T09:14:26.3268445Z /home/runner/work/nwaku/nwaku/vendor/nimbus-build-system/vendor/Nim/lib/system/chcks.nim(23) raiseIndexError2
2022-12-14T09:14:26.3268965Z /home/runner/work/nwaku/nwaku/vendor/nimbus-build-system/vendor/Nim/lib/system/fatal.nim(53) sysFatal
2022-12-14T09:14:26.3269262Z [[reraised from:
2022-12-14T09:14:26.3269902Z /home/runner/work/nwaku/nwaku/vendor/nim-json-rpc/json_rpc/client.nim(467) main
2022-12-14T09:14:26.3270350Z /home/runner/work/nwaku/nwaku/vendor/nim-json-rpc/json_rpc/client.nim(457) NimMain
2022-12-14T09:14:26.3270779Z /home/runner/work/nwaku/nwaku/vendor/nim-json-rpc/json_rpc/client.nim(448) PreMain
2022-12-14T09:14:26.3271235Z /home/runner/work/nwaku/nwaku/vendor/nim-json-rpc/json_rpc/client.nim(116) PreMainInner
2022-12-14T09:14:26.3271689Z /home/runner/work/nwaku/nwaku/vendor/nim-unittest2/unittest2.nim(848) suitePayload
2022-12-14T09:14:26.3272178Z /home/runner/work/nwaku/nwaku/vendor/nim-testutils/testutils/unittests.nim(16) runTestX60gensym680
2022-12-14T09:14:26.3272646Z /home/runner/work/nwaku/nwaku/vendor/nim-testutils/testutils/unittests.nim(15) scenario
2022-12-14T09:14:26.3273124Z /home/runner/work/nwaku/nwaku/vendor/nim-chronos/chronos/asyncfutures2.nim(391) futureContinue
2022-12-14T09:14:26.3273398Z ]]
2022-12-14T09:14:26.3273503Z 
2022-12-14T09:14:26.3273687Z     Unhandled exception: index out of bounds, the container is empty [IndexDefect]
2022-12-14T09:14:26.3274065Z   [FAILED] Admin API: connect to ad-hoc peers

Causes: not identified
TimesSeen: 1

Action type: CI
OS: linux
Test name/version: "mounting waku rln-relay: check correct registration of peers without rln-relay credentials in dynamic/on-chain mode" (v2) - happening in different tests
nwaku commit/tree: https://github.com/waku-org/nwaku/tree/226b44c86d7a0abe7a4394f2a52efba77f089c90
Log excerpt:

2022-12-14T11:47:56.8062804Z /home/runner/work/nwaku/nwaku/waku/v2/protocol/waku_relay.nim(560) main
2022-12-14T11:47:56.8063383Z /home/runner/work/nwaku/nwaku/waku/v2/protocol/waku_relay.nim(550) NimMain
2022-12-14T11:47:56.8064116Z /home/runner/work/nwaku/nwaku/waku/v2/protocol/waku_relay.nim(541) PreMain
2022-12-14T11:47:56.8064752Z /home/runner/work/nwaku/nwaku/vendor/nim-unittest2/unittest2.nim(848) suitePayload
2022-12-14T11:47:56.8065269Z /home/runner/work/nwaku/nwaku/vendor/nim-testutils/testutils/unittests.nim(16) runTestX60gensym1280
2022-12-14T11:47:56.8065795Z /home/runner/work/nwaku/nwaku/vendor/nim-chronos/chronos/asyncloop.nim(1203) waitFor
2022-12-14T11:47:56.8066254Z /home/runner/work/nwaku/nwaku/vendor/nim-chronos/chronos/asyncloop.nim(295) poll
2022-12-14T11:47:56.8066737Z /home/runner/work/nwaku/nwaku/vendor/nim-chronos/chronos/asyncfutures2.nim(369) futureContinue
2022-12-14T11:47:56.8067121Z /home/runner/work/nwaku/nwaku/tests/v2/test_waku_rln_relay_onchain.nim(572) scenario
2022-12-14T11:47:56.8067451Z /home/runner/work/nwaku/nwaku/waku/v2/node/waku_node.nim(954) start
2022-12-14T11:47:56.8067897Z /home/runner/work/nwaku/nwaku/vendor/nim-chronos/chronos/asyncfutures2.nim(369) futureContinue
2022-12-14T11:47:56.8068241Z /home/runner/work/nwaku/nwaku/waku/v2/node/waku_node.nim(984) start
2022-12-14T11:47:56.8068649Z /home/runner/work/nwaku/nwaku/vendor/nim-libp2p/libp2p/switch.nim(296) start
2022-12-14T11:47:56.8069109Z /home/runner/work/nwaku/nwaku/vendor/nim-chronos/chronos/asyncfutures2.nim(369) futureContinue
2022-12-14T11:47:56.8069548Z /home/runner/work/nwaku/nwaku/vendor/nim-libp2p/libp2p/switch.nim(311) start
2022-12-14T11:47:56.8070231Z /home/runner/work/nwaku/nwaku/vendor/nim-libp2p/libp2p/transports/tcptransport.nim(138) start
2022-12-14T11:47:56.8070717Z /home/runner/work/nwaku/nwaku/vendor/nim-chronos/chronos/asyncfutures2.nim(372) futureContinue
2022-12-14T11:47:56.8071204Z /home/runner/work/nwaku/nwaku/vendor/nim-libp2p/libp2p/transports/tcptransport.nim(157) start
2022-12-14T11:47:56.8071672Z /home/runner/work/nwaku/nwaku/vendor/nim-libp2p/libp2p/wire.nim(145) createStreamServer
2022-12-14T11:47:56.8071959Z [[reraised from:
2022-12-14T11:47:56.8072226Z /home/runner/work/nwaku/nwaku/waku/v2/protocol/waku_relay.nim(560) main
2022-12-14T11:47:56.8072565Z /home/runner/work/nwaku/nwaku/waku/v2/protocol/waku_relay.nim(550) NimMain
2022-12-14T11:47:56.8072895Z /home/runner/work/nwaku/nwaku/waku/v2/protocol/waku_relay.nim(541) PreMain
2022-12-14T11:47:56.8073324Z /home/runner/work/nwaku/nwaku/vendor/nim-unittest2/unittest2.nim(848) suitePayload
2022-12-14T11:47:56.8073805Z /home/runner/work/nwaku/nwaku/vendor/nim-testutils/testutils/unittests.nim(16) runTestX60gensym1280
2022-12-14T11:47:56.8074280Z /home/runner/work/nwaku/nwaku/vendor/nim-chronos/chronos/asyncloop.nim(1203) waitFor
2022-12-14T11:47:56.8074717Z /home/runner/work/nwaku/nwaku/vendor/nim-chronos/chronos/asyncloop.nim(295) poll
2022-12-14T11:47:56.8075177Z /home/runner/work/nwaku/nwaku/vendor/nim-chronos/chronos/asyncfutures2.nim(369) futureContinue
2022-12-14T11:47:56.8075544Z /home/runner/work/nwaku/nwaku/tests/v2/test_waku_rln_relay_onchain.nim(572) scenario
2022-12-14T11:47:56.8075874Z /home/runner/work/nwaku/nwaku/waku/v2/node/waku_node.nim(954) start
2022-12-14T11:47:56.8076308Z /home/runner/work/nwaku/nwaku/vendor/nim-chronos/chronos/asyncfutures2.nim(369) futureContinue
2022-12-14T11:47:56.8076656Z /home/runner/work/nwaku/nwaku/waku/v2/node/waku_node.nim(984) start
2022-12-14T11:47:56.8077060Z /home/runner/work/nwaku/nwaku/vendor/nim-libp2p/libp2p/switch.nim(296) start
2022-12-14T11:47:56.8077510Z /home/runner/work/nwaku/nwaku/vendor/nim-chronos/chronos/asyncfutures2.nim(372) futureContinue
2022-12-14T11:47:56.8077944Z /home/runner/work/nwaku/nwaku/vendor/nim-libp2p/libp2p/switch.nim(318) start
2022-12-14T11:47:56.8078267Z ]]
2022-12-14T11:47:56.8078440Z [[reraised from:
2022-12-14T11:47:56.8078709Z /home/runner/work/nwaku/nwaku/waku/v2/protocol/waku_relay.nim(560) main
2022-12-14T11:47:56.8079043Z /home/runner/work/nwaku/nwaku/waku/v2/protocol/waku_relay.nim(550) NimMain
2022-12-14T11:47:56.8079379Z /home/runner/work/nwaku/nwaku/waku/v2/protocol/waku_relay.nim(541) PreMain
2022-12-14T11:47:56.8079789Z /home/runner/work/nwaku/nwaku/vendor/nim-unittest2/unittest2.nim(848) suitePayload
2022-12-14T11:47:56.8080270Z /home/runner/work/nwaku/nwaku/vendor/nim-testutils/testutils/unittests.nim(16) runTestX60gensym1280
2022-12-14T11:47:56.8080733Z /home/runner/work/nwaku/nwaku/vendor/nim-chronos/chronos/asyncloop.nim(1203) waitFor
2022-12-14T11:47:56.8081209Z /home/runner/work/nwaku/nwaku/vendor/nim-chronos/chronos/asyncloop.nim(295) poll
2022-12-14T11:47:56.8081676Z /home/runner/work/nwaku/nwaku/vendor/nim-chronos/chronos/asyncfutures2.nim(369) futureContinue
2022-12-14T11:47:56.8082057Z /home/runner/work/nwaku/nwaku/tests/v2/test_waku_rln_relay_onchain.nim(572) scenario
2022-12-14T11:47:56.8082383Z /home/runner/work/nwaku/nwaku/waku/v2/node/waku_node.nim(954) start
2022-12-14T11:47:56.8082819Z /home/runner/work/nwaku/nwaku/vendor/nim-chronos/chronos/asyncfutures2.nim(372) futureContinue
2022-12-14T11:47:56.8083267Z /home/runner/work/nwaku/nwaku/vendor/nim-chronos/chronos/asyncmacro2.nim(288) start
2022-12-14T11:47:56.8083521Z ]]
2022-12-14T11:47:56.8083692Z [[reraised from:
2022-12-14T11:47:56.8083961Z /home/runner/work/nwaku/nwaku/waku/v2/protocol/waku_relay.nim(560) main
2022-12-14T11:47:56.8084289Z /home/runner/work/nwaku/nwaku/waku/v2/protocol/waku_relay.nim(550) NimMain
2022-12-14T11:47:56.8084619Z /home/runner/work/nwaku/nwaku/waku/v2/protocol/waku_relay.nim(541) PreMain
2022-12-14T11:47:56.8085032Z /home/runner/work/nwaku/nwaku/vendor/nim-unittest2/unittest2.nim(848) suitePayload
2022-12-14T11:47:56.8085557Z /home/runner/work/nwaku/nwaku/vendor/nim-testutils/testutils/unittests.nim(16) runTestX60gensym1280
2022-12-14T11:47:56.8086034Z /home/runner/work/nwaku/nwaku/vendor/nim-chronos/chronos/asyncloop.nim(1203) waitFor
2022-12-14T11:47:56.8086469Z /home/runner/work/nwaku/nwaku/vendor/nim-chronos/chronos/asyncloop.nim(295) poll
2022-12-14T11:47:56.8086920Z /home/runner/work/nwaku/nwaku/vendor/nim-chronos/chronos/asyncfutures2.nim(372) futureContinue
2022-12-14T11:47:56.8087384Z /home/runner/work/nwaku/nwaku/vendor/nim-chronos/chronos/asyncmacro2.nim(288) scenario
2022-12-14T11:47:56.8087645Z ]]
2022-12-14T11:47:56.8087812Z [[reraised from:
2022-12-14T11:47:56.8088075Z /home/runner/work/nwaku/nwaku/waku/v2/protocol/waku_relay.nim(560) main
2022-12-14T11:47:56.8088410Z /home/runner/work/nwaku/nwaku/waku/v2/protocol/waku_relay.nim(550) NimMain
2022-12-14T11:47:56.8088923Z /home/runner/work/nwaku/nwaku/waku/v2/protocol/waku_relay.nim(541) PreMain
2022-12-14T11:47:56.8089363Z /home/runner/work/nwaku/nwaku/vendor/nim-unittest2/unittest2.nim(848) suitePayload
2022-12-14T11:47:56.8089848Z /home/runner/work/nwaku/nwaku/vendor/nim-testutils/testutils/unittests.nim(16) runTestX60gensym1280
2022-12-14T11:47:56.8090135Z ]]
2022-12-14T11:47:56.8090237Z 
2022-12-14T11:47:56.8090385Z     Unhandled exception: (98) Address already in use [LPError]
2022-12-14T11:47:56.8090898Z   [FAILED] mounting waku rln-relay: check correct registration of peers without rln-relay credentials in dynamic/on-chain mode

Causes: not identified
TimesSeen: 3
Temporary workaround: it seems that re-running each failing job one at a time somehow mitigates the issue (root cause might be unrelated).

Action type: CI
OS: macOS
Test name/version: "prunePeerStore() correctly removes peers to match max quota" v2 (experimental)
nwaku commit/tree: https://github.com/waku-org/nwaku/tree/ae0a1171e605e05f03f8efeec3480bba34b62cd7/
Log excerpt:

ERR 2023-02-07 16:33:02.296+00:00 Max number of connections can't be greater than PeerManager capacity tid=67711 file=peer_manager.nim:191 capacity=5 maxConnections=10
  [OK] peer manager cant have more max connections than peerstore size
DBG 2023-02-07 16:33:02.296+00:00 no peer storage found                      tid=67711 file=peer_manager.nim:220
DBG 2023-02-07 16:33:02.297+00:00 Adding peer to manager                     topics="waku node peer_manager" tid=67711 file=peer_manager.nim:244 peerId=16U*Momc9y addresses=@[/ip4/0.0.0.0/tcp/0] proto=
DBG 2023-02-07 16:33:02.297+00:00 Adding peer to manager                     topics="waku node peer_manager" tid=67711 file=peer_manager.nim:244 peerId=16U*8H7Cd9 addresses=@[/ip4/0.0.0.0/tcp/0] proto=
DBG 2023-02-07 16:33:02.297+00:00 Adding peer to manager                     topics="waku node peer_manager" tid=67711 file=peer_manager.nim:244 peerId=16U*qdLnQt addresses=@[/ip4/0.0.0.0/tcp/0] proto=
DBG 2023-02-07 16:33:02.297+00:00 Adding peer to manager                     topics="waku node peer_manager" tid=67711 file=peer_manager.nim:244 peerId=16U*PoAaCn addresses=@[/ip4/0.0.0.0/tcp/0] proto=
DBG 2023-02-07 16:33:02.303+00:00 Adding peer to manager                     topics="waku node peer_manager" tid=67711 file=peer_manager.nim:244 peerId=16U*bX7QpZ addresses=@[/ip4/0.0.0.0/tcp/0] proto=
DBG 2023-02-07 16:33:02.303+00:00 Adding peer to manager                     topics="waku node peer_manager" tid=67711 file=peer_manager.nim:244 peerId=16U*77NSy8 addresses=@[/ip4/0.0.0.0/tcp/0] proto=
DBG 2023-02-07 16:33:02.303+00:00 Adding peer to manager                     topics="waku node peer_manager" tid=67711 file=peer_manager.nim:244 peerId=16U*sBMi5X addresses=@[/ip4/0.0.0.0/tcp/0] proto=
DBG 2023-02-07 16:33:02.303+00:00 Adding peer to manager                     topics="waku node peer_manager" tid=67711 file=peer_manager.nim:244 peerId=16U*vbsUne addresses=@[/ip4/0.0.0.0/tcp/0] proto=
DBG 2023-02-07 16:33:02.303+00:00 Adding peer to manager                     topics="waku node peer_manager" tid=67711 file=peer_manager.nim:244 peerId=16U*r5UTn9 addresses=@[/ip4/0.0.0.0/tcp/0] proto=
DBG 2023-02-07 16:33:02.303+00:00 Adding peer to manager                     topics="waku node peer_manager" tid=67711 file=peer_manager.nim:244 peerId=16U*pQGrsR addresses=@[/ip4/0.0.0.0/tcp/0] proto=
DBG 2023-02-07 16:33:02.303+00:00 Adding peer to manager                     topics="waku node peer_manager" tid=67711 file=peer_manager.nim:244 peerId=16U*twkXMB addresses=@[/ip4/0.0.0.0/tcp/0] proto=
DBG 2023-02-07 16:33:02.303+00:00 Adding peer to manager                     topics="waku node peer_manager" tid=67711 file=peer_manager.nim:244 peerId=16U*TyeLQZ addresses=@[/ip4/0.0.0.0/tcp/0] proto=
DBG 2023-02-07 16:33:02.304+00:00 Adding peer to manager                     topics="waku node peer_manager" tid=67711 file=peer_manager.nim:244 peerId=16U*HzqwTU addresses=@[/ip4/0.0.0.0/tcp/0] proto=
DBG 2023-02-07 16:33:02.304+00:00 Adding peer to manager                     topics="waku node peer_manager" tid=67711 file=peer_manager.nim:244 peerId=16U*WrBXn2 addresses=@[/ip4/0.0.0.0/tcp/0] proto=
DBG 2023-02-07 16:33:02.304+00:00 Adding peer to manager                     topics="waku node peer_manager" tid=67711 file=peer_manager.nim:244 peerId=16U*x6KKq6 addresses=@[/ip4/0.0.0.0/tcp/0] proto=
DBG 2023-02-07 16:33:02.304+00:00 Peer store capacity exceeded               topics="waku node peer_manager" tid=67711 file=peer_manager.nim:403 numPeers=15 capacity=10
DBG 2023-02-07 16:33:02.304+00:00 Finished pruning peer store                topics="waku node peer_manager" tid=67711 file=peer_manager.nim:424 beforeNumPeers=15 afterNumPeers=10 capacity=10 pruned=5
    /Users/runner/work/nwaku/nwaku/tests/v2/test_peer_manager.nim(555, 6): Check failed: not pm.peerStore.peers.anyIt(it.peerId == peers[1].peerId)
    pm.peerStore.peers.anyIt(it.peerId == peers[1].peerId) was true
  [FAILED] prunePeerStore() correctly removes peers to match max quota

Causes: not identified
TimesSeen: 1
Temporary fix/workaround: none

The text was updated successfully, but these errors were encountered:

jm-clius · 2022-11-21T13:17:13Z

@s1fr0 any update on this? Have we come to a conclusion whether this is in any way related to timing only?

LNSD · 2022-11-21T21:54:51Z

Have we come to a conclusion whether this is in any way related to timing only?

This is an umbrella issue. @jm-clius is referring to the investigation held in PR #1368.

jm-clius · 2022-11-28T13:47:51Z

I will split this into separate issues and address the Waku Bridge failures.

jm-clius · 2023-02-02T13:19:42Z

Revisiting this. Some impressions from looking at recent errors:

"Messages are bridged between Waku v1 and Waku v2" failures have either disappeared or become less since the fix.
CI: Various "test_waku_filter.nim" "Check failed: await pushHandlerFuture.withTimeout()" errors still occur only on macos tests. Will focus on improving this.
Experimental: macos tests often time out.

jm-clius · 2023-02-02T13:49:57Z

Filter test failures are all due to race conditions between client subscribing and service node handling a message that matches that subscription. I will address the timing, but this is a fundamental flaw in the protocol being addressed in: vacp2p/rfc#562

jm-clius · 2023-02-02T14:03:48Z

Draft for improving filter timeouts: #1529

jm-clius · 2023-02-27T13:05:42Z

Afaict most of the existing CI issues have been addressed and #1496 will address the remaining timeouts on macos experimental builds.

rymnc · 2023-02-28T13:39:04Z

#1496 has been merged, closing.

s1fr0 added bug Something isn't working track:maintenance labels Nov 8, 2022

oskarth added this to Vac Research and Waku Nov 8, 2022

This was referenced Nov 9, 2022

fix: move stopGanache/uninstallGanache to teardown #1364

Closed

fix(wakubridge): fix test failure while bridging messages between Waku v1 and v2 (WIP) #1368

Closed

jm-clius moved this to In Progress in Waku Nov 16, 2022

jm-clius assigned s1fr0 Nov 21, 2022

jm-clius added this to the Release 0.14.0 milestone Nov 21, 2022

This was referenced Nov 22, 2022

refactor(rln): decouple rln types from waku message type #1399

Merged

test(archive): randomize archive driver query test suites' fixtures #1411

Merged

fix(archive): patch the sqlite driver to solve results inconsistencies #1415

Merged

jm-clius assigned jm-clius and unassigned s1fr0 Nov 28, 2022

jm-clius mentioned this issue Nov 28, 2022

fix: waku bridge test timing #1429

Merged

LNSD mentioned this issue Dec 9, 2022

chore(rln-relay): refactor mounting procedure #1457

Merged

3 tasks

LNSD mentioned this issue Jan 9, 2023

bug: nwaku does not release port + does not terminate when fails to bind to a port #1478

Closed

jm-clius modified the milestones: Release 0.14.0, Release 0.15.0 Jan 9, 2023

jm-clius moved this from In Progress to Todo in Waku Jan 16, 2023

This was referenced Jan 25, 2023

refactor(archive): sqlite query optimization. replace or-ed equal conditions list with in-list #1510

Merged

fix(archive): reverse the db query results in the waku archive front-end #1511

Merged

LNSD mentioned this issue Jan 27, 2023

refactor(tests): group waku store test suite files #1515

Merged

1 task

jm-clius moved this from Todo to In Progress in Waku Jan 30, 2023

jm-clius mentioned this issue Feb 2, 2023

fix: improve filter test race condition #1529

Merged

This was referenced Feb 13, 2023

fix(tests): do not specify port number #1552

Merged

fix: remove unused imports #1553

Merged

jm-clius modified the milestones: Release 0.15.0, Release 0.16.0 Feb 27, 2023

jm-clius mentioned this issue Feb 27, 2023

feat(rln-relay): group manager integration #1496

Merged

rymnc closed this as completed Feb 28, 2023

github-project-automation bot moved this from In Progress to Done in Waku Feb 28, 2023

github-project-automation bot moved this to Done in Vac Research Feb 28, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

maintenance: failing tests #1357

maintenance: failing tests #1357

s1fr0 commented Nov 8, 2022 •

edited

Loading

jm-clius commented Nov 21, 2022

LNSD commented Nov 21, 2022 •

edited

Loading

jm-clius commented Nov 28, 2022

jm-clius commented Feb 2, 2023

jm-clius commented Feb 2, 2023

jm-clius commented Feb 2, 2023

jm-clius commented Feb 27, 2023

rymnc commented Feb 28, 2023

maintenance: failing tests #1357

maintenance: failing tests #1357

Comments

s1fr0 commented Nov 8, 2022 • edited Loading

Problem

Test report template

Failing tests

jm-clius commented Nov 21, 2022

LNSD commented Nov 21, 2022 • edited Loading

jm-clius commented Nov 28, 2022

jm-clius commented Feb 2, 2023

jm-clius commented Feb 2, 2023

jm-clius commented Feb 2, 2023

jm-clius commented Feb 27, 2023

rymnc commented Feb 28, 2023

s1fr0 commented Nov 8, 2022 •

edited

Loading

LNSD commented Nov 21, 2022 •

edited

Loading