Uploaded image for project: 'Percona XtraDB Cluster'
  1. Percona XtraDB Cluster
  2. PXC-1886

LP #1549704: Silent abort (crash) at gcs/src/gcs_core.cpp:1152

    Details

    • Type: Bug
    • Status: Done
    • Priority: Low
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None

      Description

      **Reported in Launchpad by Vladimir Stackov last update 20-05-2016 14:42:17

      Description:

      After some period of stable work Percona XtraDB Cluster nodes starts crashing and can't join cluster (no matter IST/SST).
      The only way to restore cluster is to stop all nodes and rebootstrap.

      Steps to reproduce (reproducibility - 100%):

      1. Bootstrap cluster
      2. Wait for 15-20 days
      3. Some XtraDB node will crash
      4. Try to join cluster (i.e. systemctl start mysql).

      Actual results:

      Crash and inability to join cluster.

      2016-02-05 10:42:27 6383 [Warning] WSREP: 1.0 (mysql-rw0): State transfer to 0.0 (mysql-rw1) failed: -12 (Cannot allocate memory)
      2016-02-05 10:42:27 6383 [ERROR] WSREP: gcs/src/gcs_group.cpp:gcs_group_handle_join_msg():731: Will never receive state. Need to abort.

      Expected results:

      JOINER->SYNCED

      Additional info:

      I've tried to enable core dumping but it seems that galera disables it in sources so I've tried to run mysqld manually (i.e. without mysqld_safe wrapper) with gdb attached to catch fault and get bt.

      1. rpm -qa | grep Percona
        Percona-XtraDB-Cluster-server-56-5.6.28-25.14.1.el7.x86_64
        Percona-XtraDB-Cluster-devel-56-5.6.28-25.14.1.el7.x86_64
        Percona-XtraDB-Cluster-client-56-5.6.28-25.14.1.el7.x86_64
        Percona-XtraDB-Cluster-test-56-5.6.28-25.14.1.el7.x86_64
        Percona-XtraDB-Cluster-shared-56-5.6.28-25.14.1.el7.x86_64
        Percona-XtraDB-Cluster-galera-3-debuginfo-3.14-1.rhel7.x86_64
        Percona-XtraDB-Cluster-full-56-5.6.28-25.14.1.el7.x86_64
        Percona-XtraDB-Cluster-galera-3-3.14-1.rhel7.x86_64
        Percona-XtraDB-Cluster-garbd-3-3.14-1.rhel7.x86_64
        Percona-XtraDB-Cluster-56-debuginfo-5.6.28-25.14.1.el7.x86_64
      1. cat /etc/centos-release
        CentOS Linux release 7.2.1511 (Core)
      1. hostnamectl
        Static hostname: mysql-rw2
        Chassis: container
        Virtualization: lxc-libvirt
        Operating System: CentOS Linux 7 (Core)
        CPE OS Name: cpe:/o:centos:centos:7
        Kernel: Linux 3.10.0-229.20.1.el7.x86_64
        Architecture: x86-64

        Smart Checklist

          Attachments

            Activity

              People

              • Assignee:
                krunal.bauskar Krunal Bauskar
                Reporter:
                lpjirasync lpjirasync (Inactive)
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated: