Server::check_worker_exit_status(): worker(pid=207710, id=2) abnormal exit

[baxsmaxbax]

Message was found in OpenSwoole\HTTP\Server logs:

[2024-04-18 16:21:25 $207627.0]	WARNING	Server::check_worker_exit_status(): worker(pid=207710, id=2) abnormal exit, status=0, signal=11
A bug occurred in OpenSwoole-v22.1.2, please report it.
Please submit bug report at:
>> https://github.com/openswoole/swoole-src/issues 

OS: Linux 6.1.0-18-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.76-1 (2024-02-01) x86_64
GCC_VERSION: 12.2.0
OPENSSL_VERSION: OpenSSL 3.0.11 19 Sep 2023
PHP_VERSION : 8.3.3-1+0~20240216.17+debian12~1.gbp87e37b

[nvtienlg]

we are facing many issues like this one in production servers, any updates?

`WARNING Server::check_worker_exit_status(): worker(pid=35964, id=0) abnormal exit, status=0, signal=11
A bug occurred in OpenSwoole-v22.1.2, please report it.
Please submit bug report at:

https://github.com/openswoole/swoole-src/issues

OS: Linux 6.8.0-31-generic #31-Ubuntu SMP PREEMPT_DYNAMIC Sat Apr 20 00:40:06 UTC 2024 x86_64
GCC_VERSION: 13.2.0
OPENSSL_VERSION: OpenSSL 3.0.13 30 Jan 2024
PHP_VERSION : 8.3.7`

[bitslip6]

can you post a program to reproduce this bug? Also, can you turn on core dumps and send the dump file?

[dompie]

Since 01.06.2024 we also encounter many of such issues with

[2024-06-10 08:37:42 $3441050.0]        WARNING Server::check_worker_exit_status(): worker(pid=3211089, id=113) abnormal exit, status=0, signal=11
A bug occurred in OpenSwoole-v22.0.0, please report it.
Please submit bug report at:
>> https://github.com/openswoole/swoole-src/issues 

OS: Linux 5.10.0-19-amd64 #1 SMP Debian 5.10.149-2 (2022-10-21) x86_64
GCC_VERSION: 10.2.1 20210110
OPENSSL_VERSION: OpenSSL 1.1.1n  15 Mar 2022
PHP_VERSION : 8.2.10

[2024-06-10 08:37:42 @3442662.0]        WARNING TableRow::lock(): lock process[3211089] not exists, force unlock

We did not change any code since 1. May 2024 and all of a sudden this error popped up. Any advice?

[kynx]

@bitslip6 I think I've narrowed this down to the max_request_execution_time setting:

#!/usr/bin/env php
<?php

declare(strict_types=1);

use OpenSwoole\Http\Request;
use OpenSwoole\Http\Response;
use OpenSwoole\Http\Server;

$server = new Server("0.0.0.0", 8080, Server::POOL_MODE);
$server->set([
    'max_request_execution_time' => 1,
    'daemonize'                  => false,
]);

$server->on('Request', function (Request $request, Response $response) {
    $response->end('OK');
});

$server->start();

Running the server and making a request results in the worker segfaulting and the errors reported above. If you comment out max_request_execution_time you do not see them. HTH.

[kynx]

@baxsmaxbax @nvtienlg @dompie ^^^

Can you confirm if removing max_request_execution_time stops the segfaults for you too?

[baxsmaxbax]

@baxsmaxbax @nvtienlg @dompie ^^^

Can you confirm if removing max_request_execution_time stops the segfaults for you too?

I have no this setting

[dompie]

I do not have set the setting max_request_execution_time too. In my case I suppose it has something to do with a table that is not large enough, although it only utilizes all time only about 10% of capacity/size. I use only websocket server and return an empty page on http/s request.

[hugoramosandrade]

I have the same problem when I try to use "prepare" method of OpenSwoole\Coroutine\PostgreSQL:

[2024-08-05 10:19:55 $49839.0]  WARNING Server::check_worker_exit_status(): worker(pid=49846, id=2) abnormal exit, status=0, signal=11
A bug occurred in OpenSwoole-v22.1.2, please report it.
Please submit bug report at:
>> https://github.com/openswoole/swoole-src/issues 

OS: Linux 6.5.0-44-generic #44~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue Jun 18 14:36:16 UTC 2 x86_64
GCC_VERSION: 11.4.0
OPENSSL_VERSION: OpenSSL 3.0.2 15 Mar 2022
PHP_VERSION : 8.3.9

My code looks like:

$pg = DB::getInstance(); // get a instance of OpenSwoole\Coroutine\PostgreSQL
$result = $pg->prepare("SELECT * FROM doctrine_migration_versions WHERE execution_time = $1");
$result->execute([60]);

[nvtienlg]

@kynx yes it related to max_request_execution_time
Here is gdb bt
#0 0x000057b27e189390 in ?? () #1 0x00007b84189c51f1 in php_swoole_http_request_onTimeout (timer=<optimized out>, tnode=<optimized out>) at /tmp/pear/temp/openswoole/ext-src/swoole_http_server.cc:47 #2 0x00007b8418a05a52 in std::function<void (swoole::Timer*, swoole::TimerNode*)>::operator()(swoole::Timer*, swoole::TimerNode*) const (__args#1=0x57b27e15f4d0, __args#0=<optimized out>, this=0x57b27e15f508) at /usr/include/c++/13/bits/std_function.h:591 #3 swoole::Timer::select (this=0x57b27e10d970) at /tmp/pear/temp/openswoole/src/core/timer.cc:212 #4 0x00007b8418a507f5 in std::function<void (swoole::Reactor*)>::operator()(swoole::Reactor*) const (__args#0=<optimized out>, this=0x57b27e11e3c8) at /usr/include/c++/13/bits/std_function.h:591 #5 swoole::Reactor::execute_end_callbacks (this=0x57b27e10da10, timedout=timedout@entry=false) at /tmp/pear/temp/openswoole/src/reactor/base.cc:401 #6 0x00007b8418a51bfb in swoole::ReactorEpoll::wait (this=0x57b27e0f9c10, timeo=<optimized out>) at /tmp/pear/temp/openswoole/src/reactor/epoll.cc:249 #7 0x00007b8418a76d76 in swoole::Reactor::wait (timeout=0x0, this=0x57b27e10da10) at /tmp/pear/temp/openswoole/include/swoole_reactor.h:160 #8 swoole::Server::start_event_worker (this=this@entry=0x57b27e0d0b60, worker=worker@entry=0x7b841f713ab8) at /tmp/pear/temp/openswoole/src/server/worker.cc:579 #9 0x00007b8418a577bd in swoole::Server::spawn_event_worker (this=this@entry=0x57b27e0d0b60, worker=worker@entry=0x7b841f713ab8) at /tmp/pear/temp/openswoole/src/server/manager.cc:628 #10 0x00007b8418a5852c in swoole::Manager::start (this=this@entry=0x7ffd2f4537f0, _server=_server@entry=0x57b27e0d0b60) at /tmp/pear/temp/openswoole/src/server/manager.cc:375 #11 0x00007b8418a5973b in swoole::Server::start_manager_process (this=0x57b27e0d0b60) at /tmp/pear/temp/openswoole/src/server/manager.cc:189 #12 0x00007b8418a69a99 in swoole::ProcessFactory::start (this=0x57b27e0d17b0) at /tmp/pear/temp/openswoole/src/server/process.cc:123 #13 0x00007b8418a6254a in swoole::Server::start (this=this@entry=0x57b27e0d0b60) at /tmp/pear/temp/openswoole/src/server/master.cc:579 #14 0x00007b84189e1141 in zim_swoole_server_start (execute_data=<optimized out>, return_value=0x7ffd2f453a00) at /tmp/pear/temp/openswoole/ext-src/swoole_server.cc:2563 #15 0x000057b27cd0f70a in execute_ex ()

Seem Openswoole will add internal timer to track timeout, in swoole_http_server.cc
if (serv->max_request_execution_time > 0) { swoole_timer_add((long) (serv->max_request_execution_time * 1000), false, php_swoole_http_request_onTimeout, (HttpContext *) ctx); }

Somehow php_swoole_http_request_onTimeout causes segment fault
void php_swoole_http_request_onTimeout(Timer *timer, TimerNode *tnode) { HttpContext *ctx = (HttpContext *) tnode->data; if (!ctx || (ctx->end_ || ctx->detached) || !ctx->fd) { return; } ctx->send(ctx, SW_STRL(SW_HTTP_REQUEST_TIMEOUT_PACKET)); ctx->close(ctx); }

[kynx]

@nvtienlg Good detective work! Maybe this should be raised as a new issue, since it doesn't sound like the cause of the original issue posted by @baxsmaxbax and has a reproducer and trace?