percona56-cluster-1.2 - fails to transfer state snapshot with xtrabackup (2015Q4)

[pannon]

In a nutshell cluster setup fails at the SST step (state snapshot transfer).

The following is seen in the /var/log/mysql/error.log:

WSREP_SST: [INFO] Stale sst_in_progress file: /var/mysql//sst_in_progress (20160502 09:47:12.438)
usage:  pfiles [-F] { pid | core } ...
  (report open files of each process)
  -F: force grabbing of the target process
WSREP_SST: [INFO] Evaluating timeout -k 110 100 socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20160502 09:47:12.591)
2016-05-02 09:47:12 69144 [ERROR] WSREP: Failed to read 'ready <addr>' from: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '192.168.x.x' --datadir '/var/mysql/' --defaults-file '/opt/local/etc/my.cnf' --defaults-group-suffix '' --parent '69144'  '' 
        Read: '69429:   socat -u TCP-LISTEN:4444,reuseaddr stdio'
2016-05-02 09:47:14 69144 [Note] WSREP: (42459ef3, 'tcp://0.0.0.0:4567') turning message relay requesting off

The same steps (and SST) work with percona55-cluster-1.2.

[mamash]

Sounds like a regression, I'll investigate.

[mamash]

Peter, can you help reproduce the situation where this happens, please?

[pannon]

Hi Filip, yes, did a fresh install and followed the Joyent percona guide

Same error.

What information do you need at this stage?

[dcrudgington]

For the "WSREP: Failed to read" error it looks like it may be just a PATH or can't find something issue which may be due to the pfiles. You could run "wsrep_sst_xtrabackup" manually to see what it can't find.

[pannon]

Will try to dig a bit deeper and report back.

[pannon]

Sorry for the long delay (bit busy lately..). Finally managed to track this down. The issue is in the /opt/local/bin/wsrep_sst_xtrabackup-v2 file.

There are basically two typos.

The first one is with pfiles around line 513:
should be: (pfiles $(pgrep 'socat|nc') || true) | grep "AF_INET.* ${PORT}$" >/dev/null && break
but in the latest version is missing the braces: pfiles $(pgrep 'socat|nc') || true | grep "AF_INET.* ${PORT}$" >/dev/null && break

The second is with the use of pgid (multiple occurrences).

Example, around line 406 pgid=$(ps -o pgid= $$ | grep -o '[0-9]*') this should be definitely pgid=$(ps -o pgid=$$ | grep -o '[0-9]*') (without the space pgid= $$).

But even then it will throw an error, for this I don't have a proper fix yet.

SST is working now and I have a working Percona56 cluster on SmartOS.

[pannon]

Attaching my working copy of /opt/local/bin/wsrep_sst_xtrabackup-v2.

wsrep_sst_xtrabackup-v2.zip

[mamash]

Peter, I implemented the first correction today. The ps should actually be ps -o pgid= -p $$ - omitting the -p arg works on some systems, but not on SunOS. The 2015Q4 cluster packages were just updated to 5.6.0, including this fix.

[pannon]

Thanks Filip, would you like me to test anything at this stage? I need to spin up a new cluster anyway...

[mamash]

Feel free to test away. If those two changes were all you had to make, things should work out of the box now.

[pannon]

Verified and tested, thanks again.

[alcir]

Every a few time I try to install percona cluster on smartos. And every time I am unable to put it to work.
This time I get:

2016-12-01 17:01:16 84268 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '10.96.11.215:4444/xtrabackup_sst//1' --socket '/tmp/mysql.sock' --datadir '/var/mysql/' --defaults-file '/opt/local/etc/my.cnf' --defaults-group-suffix '' '' --gtid '48aa3e0c-b7d8-11e6-be4c-d385ab82e6e9:5': 22 (Invalid argument)

SmartOS/2016Q3/x86_64

[alcir]

Wow. Increasing memory assigned to the zone, solved the issue.