main v1.7.1 [unitd --control 127.0.0.1:8300]: [alert] 28169#28169 process 28858 exited on signal 11 #221

maksitop · 2019-02-12T11:20:59Z

Hi!
messages.log
Feb 12 14:05:14 srvdb004 kernel: unitd[28863]: segfault at b8 ip 00007f9a06680c9e sp 00007f9a01bc8b00 error 4 in unitd[7f9a0665d000+49000]
Feb 12 14:05:14 srvdb004 unit: main v1.7.1 [unitd --control 127.0.0.1:8300]: [alert] 28169#28169 process 28858 exited on signal 11
unit.log
2019/02/12 14:05:14.791 [warn] 28859#28859 [unit] failed to send shm to 28858: Broken pipe (32)
2019/02/12 14:05:14.791 [warn] 28859#28859 [unit] port_send(8, 16) failed: Broken pipe (32)
2019/02/12 14:05:14.791 [alert] 28859#28859 [unit] #4: last message send failed: Broken pipe (32)

Help!

VBart · 2019-02-12T12:28:46Z

Unfortunately, these messages have no information about the problem.

Could you provide your configuration, describe your setup and conditions when you experienced this crash? Also, enabling debug logging will help: http://unit.nginx.org/troubleshooting/#debug-log

maksitop · 2019-02-12T13:28:44Z

#unitd --control 127.0.0.1:8300
#curl 127.0.0.1:8300
{
"certificates": {},
"config": {
"listeners": {
"*:8301": {
"application": "sar"
}
},

            "applications": {
                    "sar": {
                            "type": "php",
                            "processes": 2,
                            "root": "/var/www/html/",
                            "user": "nobody",
                            "group": "nobody",
                            "index": "index.php",
                            "limits": {
                                    "timeout": 10
                            }
                    }
            }
    }

}
#wget http://127.0.0.1/sar/index.htm
--2019-02-12 16:33:06-- http://127.0.0.1/sar/index.htm
Connecting to 127.0.0.1:80... connected.
HTTP request sent, awaiting response... 408 Request Timeout
2019-02-12 16:33:06 ERROR 408: Request Timeout.

Feb 12 16:31:22 srvdb004 unit: router: [alert] 29261#29265 *9 writev(-1, 2) failed (9: Bad file descriptor)
Feb 12 16:31:22 srvdb004 kernel: unitd[29265]: segfault at b8 ip 00007f9a06680c9e sp 00007f9a025c9b00 error 4 in unitd[7f9a0665d000+49000]
Feb 12 16:31:22 srvdb004 kernel: unitd[29262]: segfault at ffffffffffffffe1 ip 00007f9a037c7868 sp 00007fffcde99fb0 error 4 in php.unit.so[7f9a037c3000+a000]
Feb 12 16:31:22 srvdb004 unit: main v1.7.1 [unitd --control 127.0.0.1:8300]: [alert] 28169#28169 process 29262 exited on signal 11
Feb 12 16:31:22 srvdb004 unit: main v1.7.1 [unitd --control 127.0.0.1:8300]: [alert] 28169#28169 process 29261 exited on signal 11

2019/02/12 16:31:22 [info] 31508#31508 OpenSSL 1.0.1e-fips 11 Feb 2013, 1000105f
2019/02/12 16:31:22 [info] 31509#31509 "sar" application started
2019/02/12 16:31:22 [info] 31509#31509 (non ABS_MODE) php root: "/var/www/html"
2019/02/12 16:31:22 [info] 31510#31510 "sar" application started
2019/02/12 16:31:22 [info] 31510#31510 (non ABS_MODE) php root: "/var/www/html"

Started after update Feb 12 13:45:29 srvdb004 yum[27540]: Updated: unit-1.7.1-1.el6.ngx.x86_64
Feb 12 13:45:30 srvdb004 yum[27540]: Updated: unit-python-1.7.1-1.el6.ngx.x86_64
Feb 12 13:45:30 srvdb004 yum[27540]: Updated: unit-perl-1.7.1-1.el6.ngx.x86_64
Feb 12 13:45:30 srvdb004 yum[27540]: Updated: unit-go-1.7.1-1.el6.ngx.x86_64
Feb 12 13:45:30 srvdb004 yum[27540]: Updated: unit-php-1.7.1-1.el6.ngx.x86_64

before there was a problem
Feb 12 13:41:18 srvdb004 unit: router: [alert] 32673#798 *153 writev(-1, 2) failed (9: Bad file descriptor)
Feb 12 13:41:18 srvdb004 kernel: unitd[798]: segfault at b8 ip 00007f539b060c9e sp 00007f5395ba7b00 error 4 in unitd[7f539b03d000+49000]
Feb 12 13:41:18 srvdb004 unit: main v1.7 [unitd --control 127.0.0.1:8300]: [alert] 32670#32670 process 32673 exited on signal 11
Feb 12 13:41:18 srvdb004 unit: "example_python" application: [alert] 27399#27399 Python failed to import module "wsgi"
Feb 12 13:41:18 srvdb004 abrt: detected unhandled Python exception in 'unit'
Feb 12 13:41:18 srvdb004 unit: router: [alert] 27396#27396 failed to apply new conf
Feb 12 13:41:18 srvdb004 unit: controller: [alert] 32672#32672 failed to apply previous configuration
Feb 12 13:43:10 srvdb004 mple_python" application: [alert] 27488#27488 Python failed to import module "wsgi"
Feb 12 13:43:10 srvdb004 abrt: detected unhandled Python exception in 'unit'
Feb 12 13:43:10 srvdb004 er: [alert] 27485#27485 failed to apply new conf
Feb 12 13:43:10 srvdb004 roller: [alert] 27484#27484 failed to apply previous configuration

the configuration did not change month and worked

maksitop · 2019-02-12T16:14:54Z

debug
unit.log

VBart · 2019-02-12T16:52:34Z

@maksitop Thank you very much for the report. We are working on the fix.

VBart · 2019-02-12T18:01:36Z

@maksitop Could you show unitd --version as well?

VBart · 2019-02-12T19:15:35Z

@maksitop please try the following patch:

diff -r 6d72c218a055 -r db5b1fa66046 src/nxt_timer.c
--- a/src/nxt_timer.c   Mon Feb 11 15:42:16 2019 +0300
+++ b/src/nxt_timer.c   Tue Feb 12 22:12:09 2019 +0300
@@ -159,9 +159,9 @@ nxt_timer_change(nxt_event_engine_t *eng
 static void
 nxt_timer_changes_commit(nxt_event_engine_t *engine)
 {
-    nxt_timer_t         *timer, **add, **add_end;
+    nxt_timer_t         *timer;
     nxt_timers_t        *timers;
-    nxt_timer_change_t  *ch, *end;
+    nxt_timer_change_t  *ch, *end, *add, *add_end;
 
     timers = &engine->timers;
 
@@ -170,7 +170,7 @@ nxt_timer_changes_commit(nxt_event_engin
     ch = timers->changes;
     end = ch + timers->nchanges;
 
-    add = (nxt_timer_t **) ch;
+    add = ch;
     add_end = add;
 
     while (ch < end) {
@@ -185,7 +185,7 @@ nxt_timer_changes_commit(nxt_event_engin
 
             timer->time = ch->time;
 
-            *add_end++ = timer;
+            *add_end++ = *ch;
 
             if (!nxt_timer_is_in_tree(timer)) {
                 break;
@@ -209,7 +209,7 @@ nxt_timer_changes_commit(nxt_event_engin
     }
 
     while (add < add_end) {
-        timer = *add;
+        timer = add->timer;
 
         nxt_debug(timer->task, "timer rbtree insert: %M±%d",
                   timer->time, timer->bias);

maksitop · 2019-02-13T07:48:18Z

unitd --version

unitd --version
unit version: 1.7.1
configured as ./configure --prefix=/usr --state=/var/lib/unit --control=unix:/var/run/control.unit.sock --pid=/var/run/unit.pid --log=/var/log/unit.log --tests --openssl --modules=/usr/lib64/unit/modules --libdir=/usr/lib64 --cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -fPIC' --ld-opt='-Wl,-z,relro -Wl,-z,now -pie'

Name : unit Relocations: (not relocatable)
Version : 1.7 Vendor: Nginx Software, Inc.
Release : 1.el6.ngx Build Date: Thu 20 Dec 2018 09:47:01 PM MSK
Install Date: (not installed) Build Host: centos6-amd64-builder-builder.gnt.nginx.com
Group : System Environment/Daemons Source RPM: unit-1.7-1.el6.ngx.src.rpm
Size : 708024 License: ASL 2.0
Signature : RSA/SHA1, Thu 20 Dec 2018 09:54:57 PM MSK, Key ID abf5bd827bd9bf62
Packager : Nginx Software, Inc. https://www.nginx.com
URL : https://unit.nginx.org/
Summary : NGINX Unit

unit.repo
[unit]
name=unit repo
baseurl=https://packages.nginx.org/unit/centos/$releasever/$basearch/
gpgcheck=0
enabled=1

VBart · 2019-02-14T12:05:47Z

@maksitop Thanks. It looks like only CentOS 6 64-bit package is broken. We're working on preparing a new one. Also we've improved our release procedures to avoid this in future.

defanator · 2019-02-14T14:28:52Z

CentOS 6 x86_64 updated packages are coming soon, I'll post an update here.

Closes #221 on GitHub.

defanator · 2019-02-14T15:34:34Z

@maksitop please upgrade to the 1.7.1-2 version which has just been added to our CentOS 6 64-bit repository, and let us know how it goes.

maksitop · 2019-02-14T15:55:35Z

#yum install unit
Resolving Dependencies
--> Running transaction check
---> Package unit.x86_64 0:1.7.1-1.el6.ngx will be updated
--> Processing Dependency: unit = 1.7.1-1.el6.ngx for package: unit-go-1.7.1-1.el6.ngx.x86_64
--> Processing Dependency: unit = 1.7.1-1.el6.ngx for package: unit-php-1.7.1-1.el6.ngx.x86_64
--> Processing Dependency: unit = 1.7.1-1.el6.ngx for package: unit-perl-1.7.1-1.el6.ngx.x86_64
--> Processing Dependency: unit = 1.7.1-1.el6.ngx for package: unit-python-1.7.1-1.el6.ngx.x86_64
---> Package unit.x86_64 0:1.7.1-2.el6.ngx will be an update
--> Running transaction check
---> Package unit.x86_64 0:1.7.1-1.el6.ngx will be updated
--> Processing Dependency: unit = 1.7.1-1.el6.ngx for package: unit-perl-1.7.1-1.el6.ngx.x86_64
---> Package unit-go.x86_64 0:1.7.1-1.el6.ngx will be updated
---> Package unit-go.x86_64 0:1.7.1-2.el6.ngx will be an update
---> Package unit-php.x86_64 0:1.7.1-1.el6.ngx will be updated
---> Package unit-php.x86_64 0:1.7.1-2.el6.ngx will be an update
---> Package unit-python.x86_64 0:1.7.1-1.el6.ngx will be updated
---> Package unit-python.x86_64 0:1.7.1-2.el6.ngx will be an update
--> Finished Dependency Resolution
Error: Package: unit-perl-1.7.1-1.el6.ngx.x86_64 (@Unit)
Requires: unit = 1.7.1-1.el6.ngx
Removing: unit-1.7.1-1.el6.ngx.x86_64 (@Unit)
unit = 1.7.1-1.el6.ngx
Updated By: unit-1.7.1-2.el6.ngx.x86_64 (unit)
unit = 1.7.1-2.el6.ngx
Available: unit-0.3-1.el6.ngx.x86_64 (unit)
unit = 0.3-1.el6.ngx
Available: unit-0.4-1.el6.ngx.x86_64 (unit)
unit = 0.4-1.el6.ngx
Available: unit-0.5-1.el6.ngx.x86_64 (unit)
unit = 0.5-1.el6.ngx
Available: unit-0.6-1.el6.ngx.x86_64 (unit)
unit = 0.6-1.el6.ngx
Available: unit-0.7-1.el6.ngx.x86_64 (unit)
unit = 0.7-1.el6.ngx
Available: unit-1.0-1.el6.ngx.x86_64 (unit)
unit = 1.0-1.el6.ngx
Available: unit-1.1-1.el6.ngx.x86_64 (unit)
unit = 1.1-1.el6.ngx
Available: unit-1.2-1.el6.ngx.x86_64 (unit)
unit = 1.2-1.el6.ngx
Available: unit-1.3-1.el6.ngx.x86_64 (unit)
unit = 1.3-1.el6.ngx
Available: unit-1.4-1.el6.ngx.x86_64 (unit)
unit = 1.4-1.el6.ngx
Available: unit-1.4-2.el6.ngx.x86_64 (unit)
unit = 1.4-2.el6.ngx
Available: unit-1.5-1.el6.ngx.x86_64 (unit)
unit = 1.5-1.el6.ngx
Available: unit-1.6-1.el6.ngx.x86_64 (unit)
unit = 1.6-1.el6.ngx
Available: unit-1.7-1.el6.ngx.x86_64 (unit)
unit = 1.7-1.el6.ngx
You could try using --skip-broken to work around the problem
You could try running: rpm -Va --nofiles --nodigest

maksitop · 2019-02-14T15:56:02Z

oops

defanator · 2019-02-14T16:11:17Z

@maksitop CentOS 6 has perl 5.10 which is not supported by Unit, so we excluded that package. Please remove it and re-run the upgrade.

maksitop · 2019-02-15T09:03:25Z

#yum remove unit
Feb 15 11:20:23 srvdb004 yum[10975]: Erased: unit-go
Feb 15 11:20:23 srvdb004 yum[10975]: Erased: unit-python
Feb 15 11:20:23 srvdb004 yum[10975]: Erased: unit-php
Feb 15 11:20:23 srvdb004 yum[10975]: Erased: unit-perl
Feb 15 11:20:23 srvdb004 yum[10975]: Erased: unit

Ok
##yum install unit
#rpm -q unit
unit-1.7.1-2.el6.ngx.x86_64
Ok
#yum[11719]: Installed: unit-php-1.7.1-2.el6.ngx.x86_64
#rpm -q unit-php
unit-php-1.7.1-2.el6.ngx.x86_64
Ok
#/etc/init.d/unit start
Starting unitd: 2019/02/15 11:50:01 [info] 12827#12827 unit started
[ OK ]
#/etc/init.d/unit stop
Stopping unitd: [ OK ]

unitd --control 127.0.0.1:8300

2019/02/15 12:01:06 [info] 13326#13326 unit started

/etc/init.d/unit stop

Stopping unitd: [ OK ]

unit.log
2019/02/15 12:01:06 [notice] 13327#13327 process 13328 exited with code 0
2019/02/15 12:01:06 [info] 13330#13330 OpenSSL 1.0.1e-fips 11 Feb 2013, 1000105f
2019/02/15 12:01:57 [alert] 13327#13327 sendmsg(12, -1, 1) failed (32: Broken pipe)
2019/02/15 12:01:57 [alert] 13327#13327 sendmsg(13, -1, 1) failed (32: Broken pipe)
2019/02/15 12:01:57 [notice] 13327#13327 process 13329 exited with code 0

messages.log
Feb 15 12:01:57 srvdb004 unit: main v1.7.1 [unitd --control 127.0.0.1:8300]: [alert] 13327#13327 sendmsg(12, -1, 1) failed (32: Broken pipe)
Feb 15 12:01:57 srvdb004 unit: main v1.7.1 [unitd --control 127.0.0.1:8300]: [alert] 13327#13327 sendmsg(13, -1, 1) failed (32: Broken pipe)

VBart · 2019-02-15T09:45:50Z

@maksitop That's ok. Some alerts may appear while shutting down. They are harmless.

maksitop · 2019-02-15T10:18:50Z

thanks It works. Спасибо!

maksitop · 2019-02-15T10:24:32Z

Version unit-perl-1.7.1-2.el6.ngx.x86_64 will not be?

VBart · 2019-02-16T10:19:33Z

@maksitop Unit supports Perl versions 5.12 or later. Unfortunately, CentOS 6 provides Perl version 5.10, that was released more than 11 years ago. It has multiple incompatibilities with later versions and we cannot give any assurances if it works properly in Unit with Perl applications.

The unit-perl module on CentOS 6 has been built by mistake. You can try to build it yourself, but for multiple reasons I suggest to avoid using so old software versions.

maksitop · 2019-02-18T15:40:46Z

thanks It works. Спасибо!

In order to reduce number of operations over rb-tree and process them in batches simultaneously, all the timers changes are temporary stored in array. While processing of these changes, the same memory is also used for storing pointers to postpone timers adding. As the same block of memory has been referenced by two different types of pointers (nxt_timer_change_t * and nxt_timer_t **), some compilers may reorder operations with these pointers and produce broken code. See ticket #221 on GitHub for a particular case. Now the same "nxt_timer_change_t" structure is used in both cases. Also, reverted the -fno-strict-aliasing flag, which has been introduced in ef76227ec159 as a workaround for this issue.

VBart self-assigned this Feb 12, 2019

VBart added the z-bug 🐞 label Feb 12, 2019

nginx-hg-mirror closed this as completed in 1ba49d9 Feb 14, 2019

defanator reopened this Feb 14, 2019

nginx-hg-mirror pushed a commit that referenced this issue Feb 14, 2019

Packages: added "-fno-strict-aliasing" flag on CentOS 6 x86_64.

5581781

Closes #221 on GitHub.

mar0x mentioned this issue Feb 14, 2019

NginxUnit isn't be reconfigured and app responds with 408 #222

Closed

maksitop closed this as completed Feb 15, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

main v1.7.1 [unitd --control 127.0.0.1:8300]: [alert] 28169#28169 process 28858 exited on signal 11 #221

main v1.7.1 [unitd --control 127.0.0.1:8300]: [alert] 28169#28169 process 28858 exited on signal 11 #221

maksitop commented Feb 12, 2019

VBart commented Feb 12, 2019

maksitop commented Feb 12, 2019 •

edited

Loading

maksitop commented Feb 12, 2019 •

edited

Loading

VBart commented Feb 12, 2019

VBart commented Feb 12, 2019

VBart commented Feb 12, 2019

maksitop commented Feb 13, 2019 •

edited

Loading

VBart commented Feb 14, 2019

defanator commented Feb 14, 2019

defanator commented Feb 14, 2019

maksitop commented Feb 14, 2019

maksitop commented Feb 14, 2019

defanator commented Feb 14, 2019 •

edited

Loading

maksitop commented Feb 15, 2019

VBart commented Feb 15, 2019

maksitop commented Feb 15, 2019

maksitop commented Feb 15, 2019

VBart commented Feb 16, 2019

maksitop commented Feb 18, 2019

main v1.7.1 [unitd --control 127.0.0.1:8300]: [alert] 28169#28169 process 28858 exited on signal 11 #221

main v1.7.1 [unitd --control 127.0.0.1:8300]: [alert] 28169#28169 process 28858 exited on signal 11 #221

Comments

maksitop commented Feb 12, 2019

VBart commented Feb 12, 2019

maksitop commented Feb 12, 2019 • edited Loading

maksitop commented Feb 12, 2019 • edited Loading

VBart commented Feb 12, 2019

VBart commented Feb 12, 2019

VBart commented Feb 12, 2019

maksitop commented Feb 13, 2019 • edited Loading

VBart commented Feb 14, 2019

defanator commented Feb 14, 2019

defanator commented Feb 14, 2019

maksitop commented Feb 14, 2019

maksitop commented Feb 14, 2019

defanator commented Feb 14, 2019 • edited Loading

maksitop commented Feb 15, 2019

unitd --control 127.0.0.1:8300

/etc/init.d/unit stop

VBart commented Feb 15, 2019

maksitop commented Feb 15, 2019

maksitop commented Feb 15, 2019

VBart commented Feb 16, 2019

maksitop commented Feb 18, 2019

maksitop commented Feb 12, 2019 •

edited

Loading

maksitop commented Feb 12, 2019 •

edited

Loading

maksitop commented Feb 13, 2019 •

edited

Loading

defanator commented Feb 14, 2019 •

edited

Loading