Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dont stop IPC RPC service on broken pipe #8072

Open
fipso opened this issue Jan 17, 2025 · 0 comments
Open

Dont stop IPC RPC service on broken pipe #8072

fipso opened this issue Jan 17, 2025 · 0 comments

Comments

@fipso
Copy link

fipso commented Jan 17, 2025

Is your feature request related to a problem? Please describe.
Every time a client disconnects without closing the socket properly and there are still responses being sent from the IPC server, the service reports an error and stops itself. Also it deletes the IPC socket file.

sedge-execution-client | 13 Jan 13:22:09 | Waiting for an IPC connection... 
sedge-execution-client | 13 Jan 13:22:09 | IPC server error: System.IO.IOException: Unable to write data to the transport connection: Broken pipe.
sedge-execution-client |  ---> System.Net.Sockets.SocketException (32): Broken pipe
sedge-execution-client |    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.CreateException(SocketError error, Boolean forAsyncThrow)
sedge-execution-client |    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.SendAsyncForNetworkStream(Socket socket, CancellationToken cancellationToken)
sedge-execution-client |    at System.Net.Sockets.NetworkStream.WriteAsync(ReadOnlyMemory`1 buffer, CancellationToken cancellationToken)
sedge-execution-client |    at Nethermind.Serialization.Json.CountingStreamPipeWriter.FlushAsyncInternal(Boolean writeToStream, ReadOnlyMemory`1 data, CancellationToken cancellationToken) in /src/Nethermind/Nethermind.Serialization.Json/StreamPipeWriter.cs:line 366
sedge-execution-client |    at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
sedge-execution-client |    at Nethermind.Serialization.Json.CountingStreamPipeWriter.FlushAsyncInternal(Boolean writeToStream, ReadOnlyMemory`1 data, CancellationToken cancellationToken)
sedge-execution-client |    at Nethermind.Serialization.Json.CountingStreamPipeWriter.CompleteAsync(Exception exception) in /src/Nethermind/Nethermind.Serialization.Json/StreamPipeWriter.cs:line 292
sedge-execution-client |    at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
sedge-execution-client |    at Nethermind.Serialization.Json.CountingStreamPipeWriter.CompleteAsync(Exception exception)
sedge-execution-client |    at Nethermind.Serialization.Json.EthereumJsonSerializer.SerializeAsync[T](Stream stream, T value, Boolean indented, Boolean leaveOpen) in /src/Nethermind/Nethermind.Serialization.Json/EthereumJsonSerializer.cs:line 147
sedge-execution-client |    at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
sedge-execution-client |    at Nethermind.Serialization.Json.EthereumJsonSerializer.SerializeAsync[T](Stream stream, T value, Boolean indented, Boolean leaveOpen)
sedge-execution-client |    at Nethermind.JsonRpc.WebSockets.JsonRpcSocketsClient`1.SendJsonRpcResult(JsonRpcResult result) in /src/Nethermind/Nethermind.JsonRpc/WebSockets/JsonRpcSocketsClient.cs:line 156
sedge-execution-client |    at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
sedge-execution-client |    at Nethermind.JsonRpc.WebSockets.JsonRpcSocketsClient`1.SendJsonRpcResult(JsonRpcResult result)
sedge-execution-client |    at Nethermind.JsonRpc.WebSockets.JsonRpcSocketsClient`1.ProcessAsync(ArraySegment`1 data) in /src/Nethermind/Nethermind.JsonRpc/WebSockets/JsonRpcSocketsClient.cs:line 67
sedge-execution-client |    at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
sedge-execution-client |    at Nethermind.JsonRpc.WebSockets.JsonRpcSocketsClient`1.ProcessAsync(ArraySegment`1 data)
sedge-execution-client |    at Nethermind.Sockets.SocketClient`1.ReceiveLoopAsync() in /src/Nethermind/Nethermind.Sockets/SocketClient.cs:line 66
--
sedge-execution-client |    at System.Threading.Tasks.Task`1.TrySetResult(TResult result)
sedge-execution-client |    at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
sedge-execution-client |    at System.Net.Sockets.SocketAsyncEventArgs.TransferCompletionCallbackCore(Int32 bytesTransferred, Memory`1 socketAddress, SocketFlags receivedFlags, SocketError socketError)
sedge-execution-client |    at System.Net.Sockets.SocketAsyncEngine.System.Threading.IThreadPoolWorkItem.Execute()
sedge-execution-client |    at System.Threading.ThreadPoolWorkQueue.Dispatch()
sedge-execution-client |    at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
sedge-execution-client | --- End of stack trace from previous location ---
sedge-execution-client | 
sedge-execution-client |    --- End of inner exception stack trace ---
sedge-execution-client |    at Nethermind.Serialization.Json.CountingStreamPipeWriter.FlushAsyncInternal(Boolean writeToStream, ReadOnlyMemory`1 data, CancellationToken cancellationToken) in /src/Nethermind/Nethermind.Serialization.Json/StreamPipeWriter.cs:line 366
sedge-execution-client |    at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
sedge-execution-client |    at Nethermind.Serialization.Json.CountingStreamPipeWriter.CompleteAsync(Exception exception) in /src/Nethermind/Nethermind.Serialization.Json/StreamPipeWriter.cs:line 292
sedge-execution-client |    at Nethermind.Serialization.Json.CountingStreamPipeWriter.CompleteAsync(Exception exception) in /src/Nethermind/Nethermind.Serialization.Json/StreamPipeWriter.cs:line 301
sedge-execution-client |    at Nethermind.Serialization.Json.EthereumJsonSerializer.SerializeAsync[T](Stream stream, T value, Boolean indented, Boolean leaveOpen) in /src/Nethermind/Nethermind.Serialization.Json/EthereumJsonSerializer.cs:line 147
sedge-execution-client |    at Nethermind.JsonRpc.WebSockets.JsonRpcSocketsClient`1.SendJsonRpcResult(JsonRpcResult result) in /src/Nethermind/Nethermind.JsonRpc/WebSockets/JsonRpcSocketsClient.cs:line 156
sedge-execution-client |    at Nethermind.JsonRpc.WebSockets.JsonRpcSocketsClient`1.ProcessAsync(ArraySegment`1 data) in /src/Nethermind/Nethermind.JsonRpc/WebSockets/JsonRpcSocketsClient.cs:line 67
sedge-execution-client |    at Nethermind.JsonRpc.WebSockets.JsonRpcSocketsClient`1.ProcessAsync(ArraySegment`1 data) in /src/Nethermind/Nethermind.JsonRpc/WebSockets/JsonRpcSocketsClient.cs:line 63
sedge-execution-client |    at Nethermind.Sockets.SocketClient`1.ReceiveLoopAsync() in /src/Nethermind/Nethermind.Sockets/SocketClient.cs:line 66
sedge-execution-client |    at Nethermind.Sockets.SocketClient`1.ReceiveLoopAsync() in /src/Nethermind/Nethermind.Sockets/SocketClient.cs:line 96
sedge-execution-client |    at Nethermind.Runner.JsonRpc.JsonRpcIpcRunner.StartServer(String path, CancellationToken cancellationToken) in /src/Nethermind/Nethermind.Runner/JsonRpc/JsonRpcIpcRunner.cs:line 89
sedge-execution-client | 13 Jan 13:22:09 | IPC JSON RPC service stopped 

Describe the solution you'd like
Only log the broken pipe error but dont stop the whole service.

Describe alternatives you've considered
Only workaround I see is using some reverse proxy for the ipc socket that always closes sessions properly

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant