Uploaded image for project: 'Percona XtraDB Cluster'
  1. Percona XtraDB Cluster
  2. PXC-1816

LP #1441762: IST Fails with sst script error

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Priority: Low
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

      **Reported in Launchpad by Jay Janssen last update 03-06-2015 15:39:22

      This problem happens intermittently. Once it happens on a given joiner, I can't force it to IST, the node must SST. I can reproduce this with a few attempts at stopping and restarting a node while workload runs on the rest of the cluster.

      This is on Centos 7 with the latest packages:

      [root@node3 mysql]# rpm -qa | grep -i percona
      percona-toolkit-2.2.13-1.noarch
      percona-xtrabackup-2.2.10-1.el7.x86_64
      percona-release-0.1-3.noarch
      Percona-XtraDB-Cluster-shared-56-5.6.22-25.8.978.el7.x86_64
      Percona-XtraDB-Cluster-client-56-5.6.22-25.8.978.el7.x86_64
      Percona-XtraDB-Cluster-galera-3-3.9-1.3494.rhel7.x86_64
      Percona-XtraDB-Cluster-server-56-5.6.22-25.8.978.el7.x86_64
      Percona-XtraDB-Cluster-devel-56-5.6.22-25.8.978.el7.x86_64

      Log:

      2015-04-08 17:20:46 4417 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 3128)
      2015-04-08 17:20:46 4417 [Note] WSREP: State transfer required:
      Group state: b9a78dfb-de0f-11e4-9826-eb80bdd55bc5:3128
      Local state: b9a78dfb-de0f-11e4-9826-eb80bdd55bc5:705
      2015-04-08 17:20:46 4417 [Note] WSREP: New cluster view: global state: b9a78dfb-de0f-11e4-9826-eb80bdd55bc5:3128, view# 9: Prima
      ry, number of nodes: 3, my index: 2, protocol version 3
      2015-04-08 17:20:46 4417 [Warning] WSREP: Gap in state sequence. Need state transfer.
      2015-04-08 17:20:46 4417 [Note] WSREP: Running: 'wsrep_sst_xtrabackup-v2 --role 'joiner' --address '192.168.70.4' --auth 'sst:se
      cret' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --parent '4417' '' '
      WSREP_SST: [INFO] Streaming with xbstream (20150408 17:20:47.133)
      WSREP_SST: [INFO] Using socat as streamer (20150408 17:20:47.136)
      WSREP_SST: [INFO] Stale sst_in_progress file: /var/lib/mysql//sst_in_progress (20150408 17:20:47.138)
      2015-04-08 17:20:47 4417 [Note] WSREP: Prepared SST request: xtrabackup-v2|192.168.70.4:4444/xtrabackup_sst//1
      2015-04-08 17:20:47 4417 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      2015-04-08 17:20:47 4417 [Note] WSREP: REPL Protocols: 7 (3, 2)
      2015-04-08 17:20:47 4417 [Note] WSREP: Service thread queue flushed.
      2015-04-08 17:20:47 4417 [Note] WSREP: Assign initial position for certification: 3128, protocol version: 3
      2015-04-08 17:20:47 4417 [Note] WSREP: Service thread queue flushed.
      2015-04-08 17:20:47 4417 [Note] WSREP: Prepared IST receiver, listening at: tcp://192.168.70.4:4568
      WSREP_SST: [INFO] Evaluating timeout -k 110 100 socat -u TCP-LISTEN:4444,reuseaddr stdio | xbstream -x; RC=( ${PIPESTATUS[@]} ) (20150408 17:20:47.176)
      2015-04-08 17:20:47 4417 [Note] WSREP: Member 2.0 (node3) requested state transfer from 'any'. Selected 1.0 (node1)(SYNCED) as donor.
      2015-04-08 17:20:47 4417 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 3131)
      2015-04-08 17:20:47 4417 [Note] WSREP: Requesting state transfer: success, donor: 1
      2015-04-08 17:20:47 4417 [Note] WSREP: 1.0 (node1): State transfer to 2.0 (node3) complete.
      2015-04-08 17:20:47 4417 [Note] WSREP: Member 1.0 (node1) synced with group.
      WSREP_SST: [ERROR] Removing /tmp/tmp.vOO1WbEZ4X/xtrabackup_galera_info file due to signal (20150408 17:20:47.590)
      WSREP_SST: [INFO] xtrabackup_ist received from donor: Running IST (20150408 17:20:47.592)
      /usr//bin/wsrep_sst_xtrabackup-v2: line 540: pushd: /var/lib/mysql//.sst: No such file or directory
      cat: /tmp/tmp.vOO1WbEZ4X/xtrabackup_galera_info: No such file or directory
      WSREP_SST: [INFO] Galera co-ords from recovery: (20150408 17:20:47.595)
      cat: /tmp/tmp.vOO1WbEZ4X/xtrabackup_galera_info: No such file or directory
      WSREP_SST: [ERROR] Cleanup after exit with status:1 (20150408 17:20:47.597)
      2015-04-08 17:20:47 4417 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '192.168.70.4' --auth 'sst:secret' --datadir '/var/lib/mysql/' --defaults-file '/etc/my.cnf' --parent '4417' '' : 1 (Operation not permitted)
      2015-04-08 17:20:47 4417 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
      2015-04-08 17:20:47 4417 [ERROR] WSREP: SST failed: 1 (Operation not permitted)
      2015-04-08 17:20:47 4417 [ERROR] Aborting

        Smart Checklist

          Attachments

            Activity

              People

              • Assignee:
                krunal.bauskar Krunal Bauskar
                Reporter:
                lpjirasync lpjirasync (Inactive)
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated: