crash on orbiter node with v0.27.2

[packetstracer]

Our node crashed suddenly after running for several hours after version update to v0.27.2

Node logs

2022-11-09 18:26:19 [Relaychain] 💤 Idle (50 peers), best: #15255255 (0xc5e5…b0d6), finalized #15255252 (0x59af…7619), ⬇ 750.9kiB/s ⬆ 698.7kiB/s
2022-11-09 18:26:25 [Relaychain] 💤 Idle (50 peers), best: #15255255 (0xc5e5…b0d6), finalized #15255252 (0x59af…7619), ⬇ 552.5kiB/s ⬆ 83.1kiB/s
2022-11-09 18:26:29 failed to associate send_message response to the sender                             
====================                                                                                      
Version: 0.27.2-d3b7c941cf8                                                                               
   0: sp_panic_handler::set::{{closure}}                                                                                                                                                                             
   1: std::panicking::rust_panic_with_hook                                                                                                                                                                           
             at /rustc/1d37ed661a6922e7a167609b8cd7eb31e972b19b/library/std/src/panicking.rs:702:17
   2: std::panicking::begin_panic::{{closure}}                                                            
   3: std::sys_common::backtrace::__rust_end_short_backtrace            
   4: std::panicking::begin_panic                                                                                                                                                                                    
   5: <quicksink::SinkImpl<S,F,T,A,E> as futures_sink::Sink<A>>::poll_ready
   6: <libp2p::bandwidth::BandwidthConnecLogging<TInner> as futures_io::if_std::AsyncWrite>::poll_write                                                                                                              
   7: <libp2p_noise::io::framed::NoiseFramed<T,S> as futures_sink::Sink<&alloc::vec::Vec<u8>>>::poll_ready 
   8: <libp2p_noise::io::framed::NoiseFramed<T,S> as futures_sink::Sink<&alloc::vec::Vec<u8>>>::poll_flush                         
   9: <libp2p_noise::io::NoiseOutput<T> as futures_io::if_std::AsyncWrite>::poll_flush                                                                                                                               
  10: multistream_select::negotiated::Negotiated<TInner>::poll                
  11: <multistream_select::negotiated::Negotiated<TInner> as futures_io::if_std::AsyncWrite>::poll_close
  12: yamux::connection::Connection<T>::next::{{closure}}                                                                                                                                                            
  13: <futures_util::stream::try_stream::ErrInto<St,E> as futures_core::stream::Stream>::poll_next       
  14: <libp2p_core::muxing::boxed::Wrap<T> as libp2p_core::muxing::StreamMuxer>::poll_close                                                                                                                          
  15: <futures_util::future::poll_fn::PollFn<F> as core::future::future::Future>::poll
  16: tokio::runtime::task::core::CoreStage<T>::poll                                                                                                                                                                 
  17: tokio::runtime::task::raw::poll                                                                                                                                                                                
  18: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
  19: tokio::runtime::scheduler::multi_thread::worker::run                                                                                                                                                           
  20: tokio::runtime::task::raw::poll                                                                     
  21: std::sys_common::backtrace::__rust_begin_short_backtrace                                                                                                                                                       
  22: core::ops::function::FnOnce::call_once{{vtable.shim}}
  23: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once                                                                                                                                       
             at /rustc/1d37ed661a6922e7a167609b8cd7eb31e972b19b/library/alloc/src/boxed.rs:1940:9
      <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once                                                                                                                                       
             at /rustc/1d37ed661a6922e7a167609b8cd7eb31e972b19b/library/alloc/src/boxed.rs:1940:9                                                                                                                    
      std::sys::unix::thread::Thread::new::thread_start                                                                                                                                                              
             at /rustc/1d37ed661a6922e7a167609b8cd7eb31e972b19b/library/std/src/sys/unix/thread.rs:108:17
  24: start_thread                                                                                                                                                                                                   
  25: clone                                                                                                                                                                                                          
Thread 'tokio-runtime-worker' panicked at 'SinkImpl::poll_ready called after error.', /home/gh-actions/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/quicksink-0.1.2/src/lib.rs:158
This is a bug. Please report it at:                                                                       
        https://github.com/PureStake/moonbeam/issues/new                                                                                                                                                             
moonriver.service: Main process exited, code=killed, status=4/ILL                      
moonriver.service: Failed with result 'signal'.                                                                                                                                                                      moonriver.service: Scheduled restart job, restart counter is at 2.                                                                                                                                                   
Stopped "Moonriver daemon".                                                                                                                                                                                          
Started "Moonriver daemon".                                                                                                                                                                                          
`--state-cache-size` was deprecated. Please switch to `--trie-cache-size`.                                                                                                                                           
2022-11-09 18:28:08 Moonbeam Parachain Collator                                                           
2022-11-09 18:28:08 ✌️  version 0.27.2-d3b7c941cf8                                                                                                                                                                    
2022-11-09 18:28:08 ❤️  by PureStake, 2019-2022                                                            
2022-11-09 18:28:08 📋 Chain specification: Moonriver 

Startup command

/usr/local/bin/moonbeam-znver3 --base-path=/home/blockscope/moonriver-data --chain moonriver \
--name="Blockscope" \
--validator \
--execution wasm \
--wasm-execution compiled \
--pruning archive \
--state-cache-size 1 \
--db-cache 32000 \
--port 30333 \
--rpc-port 9933 \
--ws-port 9944 \
-- \
--execution wasm \
--port 30334 \
--rpc-port 9934 \
--ws-port 9945 \
--pruning archive \
--name="Blockscope (Relay)"

[packetstracer]

we've observed that happened also with other versions, so not related to latest version.

Trying with other server.

[crystalin]

Thank you for the report:
@bkchr have you seen this error before ?

[bkchr]

@crystalin try cargo update -p yamux. That should fix it

[librelois]

@packetstracer your problem seems related to this bug: libp2p/rust-libp2p#2598

This bug has been fixed in yamux 0.10.2 (https://github.com/libp2p/rust-yamux/blob/v0.10.2/CHANGELOG.md#0102).

The new moonbeam client v0.28.1 included this update of yamux, so the problem should be solved. If it's not the case, feel free to reopen this issue.

[packetstracer]

@packetstracer your problem seems related to this bug: libp2p/rust-libp2p#2598

This bug has been fixed in yamux 0.10.2 (https://github.com/libp2p/rust-yamux/blob/v0.10.2/CHANGELOG.md#0102).

The new moonbeam client v0.28.1 included this update of yamux, so the problem should be solved. If it's not the case, feel free to reopen this issue.

Updating to that version, reporting any issues. Thanks.