Uploaded image for project: 'Percona XtraDB Cluster'
  1. Percona XtraDB Cluster
  2. PXC-2001

LP #1712374: PXC Cluster is not writeable for a while when a node leaves/joins a network multiple times in a minute

    Details

    • Type: Bug
    • Status: Done
    • Priority: Low
    • Resolution: Cannot Reproduce
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

      **Reported in Launchpad by weishan last update 21-11-2017 04:17:25

      OS Version: RHEL 7
      PXC Version: 5.7.16-27.19.1
      Architecture: 3 node cluster. 2 nodes(nodeA & nodeB) in DC A and 1 node(nodeC) in DC B. There's a VIP controlled by Keepalived between the 2 nodes in DC A

      We had a network glitch between DC A and DC B that cause the network to break intermittently. This caused the cluster to do a "leader election" and leave the cluster un-writeable temporarily. However, we expected the 2 nodes in DC A to be up and running without any issues as they still form the majority of the cluster.

      During the time when the cluster is un-writable, we can see this in the logs.

      [Warning] WSREP: Quorum: No node with complete state:

      Version : 4
      Flags : 0x3
      Protocols : 0 / 7 / 3
      State : NON-PRIMARY
      Desync count : 0
      Prim state : SYNCED
      Prim UUID : b3f64e9a-85c0-11e7-a959-af07893cf6d6
      Prim seqno : 252
      First seqno : 86650
      Last seqno : 96481
      Prim JOINED : 2
      State UUID : becab943-85c0-11e7-8206-8b2b2ee4edcf
      Group UUID : 87995989-866b-11e6-bee8-db3321b1f4c3
      Name : 'NodeA'
      Incoming addr: 'NodeA:3306'

      Version : 4
      Flags : 0x2
      Protocols : 0 / 7 / 3
      State : NON-PRIMARY
      Desync count : 0
      Prim state : SYNCED
      Prim UUID : b3f64e9a-85c0-11e7-a959-af07893cf6d6
      Prim seqno : 252
      First seqno : 86635
      Last seqno : 96481
      Prim JOINED : 2
      State UUID : becab943-85c0-11e7-8206-8b2b2ee4edcf
      Group UUID : 87995989-866b-11e6-bee8-db3321b1f4c3
      Name : 'NodeB'
      Incoming addr: 'NodeB:3306'

      2017-08-20T16:00:53.926887Z 0 [Note] WSREP: Full re-merge of primary b3f64e9a-85c0-11e7-a959-af07893cf6d6 found: 2 of 2.
      2017-08-20T16:00:53.926894Z 0 [Note] WSREP: Quorum results:
      version = 4,
      component = PRIMARY,
      conf_id = 252,
      members = 2/2 (joined/total),
      act_id = 96481,
      last_appl. = 96376,
      protocols = 0/7/3 (gcs/repl/appl),
      group UUID = 87995989-866b-11e6-bee8-db3321b1f4c3

      Regards,
      Wei Shan

        Smart Checklist

          Attachments

            Activity

              People

              • Assignee:
                krunal.bauskar Krunal Bauskar
                Reporter:
                lpjirasync lpjirasync (Inactive)
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated: