Uploaded image for project: 'Percona XtraDB Cluster'
  1. Percona XtraDB Cluster
  2. PXC-2254

Conflict causes crash in version 5.7.23-23

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Priority: Medium
    • Resolution: Fixed
    • Affects Version/s: 5.7.23-31.31
    • Fix Version/s: 5.7.23-31.31.2
    • Component/s: None
    • Labels:
    • Environment:

      Description

      Hi,

      We upgraded from version 5.7.18 to 5.7.23 and since then a particular conflict causes the node to crash.

      We are using debian 8, and the following packages from the http://repo.percona.com/apt/dists/jessie/main repo:

      percona-xtradb-cluster-57 5.7.23-31.31-1.jesse amd64
      percona-xtradb-cluster-clent-5.7 5.7.23-31.31-1.jesse amd64
      percona-xtradb-cluster-common-5.7 5.7.23-31.31-1.jesse amd64
      percona-xtradb-cluster-server-5.7 5.7.23-31.31-1.jesse amd64

      The following log is an example of the crash:

      *** Priority TRANSACTION:
      TRANSACTION 52354503, ACTIVE 0 sec starting index read
      mysql tables in use 1, locked 1
      MySQL thread id 11, OS thread handle 139707763140352, query id 2324120 System lock*** Victim TRANSACTION:
      TRANSACTION 52354502, ACTIVE 0 sec
      , undo log entries 2
      MySQL thread id 110, OS thread handle 139657239574272, query id 2324118 192.168.100.20 prod wsrep: initiating replication for write set (-1)
      commit
      *** WAITING FOR THIS LOCK TO BE GRANTED:
      RECORD LOCKS space id 543 page no 3 n bits 88 index PRIMARY of table `database_name`.`table_name` trx id 52354502 lock_mode X
      2018-09-27T06:03:12.455655Z 11 [Note] WSREP: --------- CONFLICT DETECTED --------
      2018-09-27T06:03:12.455665Z 11 [Note] WSREP: cluster conflict due to high priority abort for threads:2018-09-27T06:03:12.455673Z 11 [Note] WSREP: Winning thread:
      {{ THD: 11, mode: applier, state: executing, conflict: no conflict, seqno: 23527460}}
      {{ SQL: (null)}}2018-09-27T06:03:12.455680Z 11 [Note] WSREP: Victim thread:
      {{ THD: 110, mode: local, state: committing, conflict: no conflict, seqno: -1}}
      {{ SQL: commit}}2018-09-27T06:03:12.458620Z 110 [ERROR] WSREP: FSM: no such a transition MUST_CERT_AND_REPLAY -> ROLLED_BACK
      06:03:12 UTC - mysqld got signal 6 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
      Attempting to collect some information that could help diagnose the problem.
      As this is a crash and something is definitely wrong, the information
      collection process might fail.
      Please help us make Percona XtraDB Cluster better by reporting any
      bugs at https://jira.percona.com/projects/PXC/issueskey_buffer_size=8388608
      read_buffer_size=131072
      max_used_connections=85
      max_threads=601
      thread_count=113
      connection_count=64
      It is possible that mysqld could use up to
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 247423 K bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.Thread pointer: 0x7ef7f9125000
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 7f047c27aa70 thread_stack 0x40000
      /usr/sbin/mysqld(my_print_stacktrace+0x2c)[0xed405c]
      /usr/sbin/mysqld(handle_fatal_signal+0x479)[0x7a5379]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0xf890)[0x7f10ce534890]
      /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37)[0x7f10cc4b9067]
      /lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7f10cc4ba448]
      /usr/lib/galera3/libgalera_smm.so(ZN6galera3FSMINS_9TrxHandle5StateENS1_10TransitionENS_10EmptyGuardENS_11EmptyActionEE8shift_toES2+0x199)[0x7f10c0576589]
      /usr/lib/galera3/libgalera_smm.so(_ZN6galera13ReplicatorSMM13post_rollbackEPNS_9TrxHandleE+0x21)[0x7f10c0569af1]
      /usr/lib/galera3/libgalera_smm.so(galera_post_rollback+0x66)[0x7f10c05854f6]
      /usr/sbin/mysqld[0xd6f3aa]
      /usr/sbin/mysqld(_Z21trans_commit_implicitP3THD+0xa0)[0xd44050]
      /usr/sbin/mysqld(_Z21ha_enable_transactionP3THDb+0x4d)[0x822a6d]
      /usr/sbin/mysqld(_ZN3THD16init_for_queriesEP14Relay_log_info+0xa1)[0xc47d81]
      /usr/sbin/mysqld[0x7bff21]
      /usr/sbin/mysqld(_Z24wsrep_replay_transactionP3THD+0x1be)[0x7c10fe]
      /usr/sbin/mysqld[0xc94aac]
      /usr/sbin/mysqld(_Z16dispatch_commandP3THDPK8COM_DATA19enum_server_command+0x1216)[0xc95ed6]
      /usr/sbin/mysqld(_Z10do_commandP3THD+0x288)[0xc973e8]
      /usr/sbin/mysqld(handle_connection+0x398)[0xd62ed8]
      /usr/sbin/mysqld(pfs_spawn_thread+0x1b4)[0xeec074]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x8064)[0x7f10ce52d064]
      /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f10cc56c62d]Trying to get some variables.
      Some pointers may be invalid and cause the dump to abort.
      Query (7ef7f9133030): is an invalid pointer
      Connection ID (thread ID): 110
      Status: NOT_KILLED

      An example of the behaviour before would be a conflict but without the crash:

      *** Priority TRANSACTION:
      TRANSACTION 49020253, ACTIVE 0 sec starting index read
      mysql tables in use 1, locked 1
      MySQL thread id 32, OS thread handle 139771241555712, query id 1088204812 System lock*** Victim TRANSACTION:
      TRANSACTION 49020252, ACTIVE (PREPARED) 0 sec
      , undo log entries 2
      MySQL thread id 600257, OS thread handle 139683014104832, query id 1088204810 192.168.100.20 prod wsrep: initiating pre-commit for write set (22215121)
      commit
      *** WAITING FOR THIS LOCK TO BE GRANTED:
      RECORD LOCKS space id 543 page no 3 n bits 200 index PRIMARY of table `database_name`.`table_name` trx id 49020252 lock_mode X
      2018-09-02T17:22:07.879938Z 32 [Note] WSREP: --------- CONFLICT DETECTED --------
      2018-09-02T17:22:07.879949Z 32 [Note] WSREP: cluster conflict due to high priority abort for threads:2018-09-02T17:22:07.879957Z 32 [Note] WSREP: Winning thread:
      {{ THD: 32, mode: applier, state: executing, conflict: no conflict, seqno: 22215118}}
      {{ SQL: (null)}}2018-09-02T17:22:07.879965Z 32 [Note] WSREP: Victim thread:
      {{ THD: 600257, mode: local, state: committing, conflict: no conflict, seqno: 22215121}}
      {{ SQL: commit}}

      In this version (5.7.18) the node wouldn't crash. Is this intentional behaviour in the new version or a bug?

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                krunal.bauskar Krunal Bauskar
                Reporter:
                laurenceg Laurence Gil
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Time Spent - 1 day, 32 minutes Remaining Estimate - 3 hours
                  3h
                  Logged:
                  Time Spent - 1 day, 32 minutes Remaining Estimate - 3 hours
                  1d 32m