Uploaded image for project: 'Percona XtraDB Cluster'
  1. Percona XtraDB Cluster
  2. PXC-3248

PXC 8 crashes on startup when upgrading from 5.7

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: On Hold
    • Priority: Medium
    • Resolution: Unresolved
    • Affects Version/s: 8.0.18-9.3
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

       

      2020-05-12T16:54:40.776149Z 0 [Warning] [MY-000000] [WSREP] Option --wsrep-causal-reads is deprecated. Please start using --wsrep-sync-wait.
      2020-05-12T16:54:40.776155Z 0 [Warning] [MY-000000] [WSREP] --wsrep-causal-reads=ON takes precedence over --wsrep-sync-wait=0. WSREP_SYNC_WAIT_BEFORE_READ is on
      2020-05-12T16:54:40.777245Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.18-9) starting as process 1583
      2020-05-12T16:54:40.778935Z 0 [Note] [MY-000000] [Galera] Loading provider /usr/lib/galera4/libgalera_smm.so initial position: 00000000-0000-0000-0000-000000000000:-1
      2020-05-12T16:54:40.778968Z 0 [Note] [MY-000000] [Galera] wsrep_load(): loading provider library '/usr/lib/galera4/libgalera_smm.so'
      2020-05-12T16:54:40.780326Z 0 [Note] [MY-000000] [Galera] wsrep_load(): Galera 4.3(r752664d) by Codership Oy <info@codership.com> loaded successfully.
      2020-05-12T16:54:40.780364Z 0 [Note] [MY-000000] [Galera] CRC-32C: using hardware acceleration.
      2020-05-12T16:54:40.780779Z 0 [Note] [MY-000000] [Galera] Found saved state: db204094-9470-11ea-9032-4277165867e8:-1, safe_to_bootstrap: 1
      2020-05-12T16:54:40.780885Z 0 [Note] [MY-000000] [Galera] GCache DEBUG: opened preamble:
      Version: 2
      UUID: db204094-9470-11ea-9032-4277165867e8
      Seqno: -1 - -1
      Offset: -1
      Synced: 0
      2020-05-12T16:54:40.780945Z 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: version: 2, UUID: db204094-9470-11ea-9032-4277165867e8, offset: -1
      2020-05-12T16:54:40.781030Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer initial scan... 0.0% ( 0/2147483672 bytes) complete.
      2020-05-12T16:54:40.781087Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer initial scan...100.0% (2147483672/2147483672 bytes) complete.
      2020-05-12T16:54:40.781153Z 0 [Note] [MY-000000] [Galera] Recovering GCache ring buffer: found gapless sequence 1-1
      2020-05-12T16:54:40.781202Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer unused buffers scan... 0.0% ( 0/168 bytes) complete.
      2020-05-12T16:54:40.781237Z 0 [Note] [MY-000000] [Galera] GCache::RingBuffer unused buffers scan...100.0% (168/168 bytes) complete.
      2020-05-12T16:54:40.781271Z 0 [Note] [MY-000000] [Galera] GCache DEBUG: RingBuffer::recover(): found 0/1 locked buffers
      2020-05-12T16:54:40.781307Z 0 [Note] [MY-000000] [Galera] GCache DEBUG: RingBuffer::recover(): used space: 168/2147483648
      2020-05-12T16:54:40.785202Z 0 [Note] [MY-000000] [Galera] Passing config to GCS: base_dir = /data/mysql/; base_host = 10.233.120.174; base_port = 4567; cert.log_conflicts = no; cert.optimistic_pa = no; debug = no; evs.auto_evict = 0; evs.delay_margin = PT1S; evs.delayed_keep_period = PT30S; evs.inactive_check_period = PT0.5S; evs.inactive_timeout = PT15S; evs.join_retrans_period = PT1S; evs.max_install_timeouts = 3; evs.send_window = 10; evs.stats_report_period = PT1M; evs.suspect_timeout = PT5S; evs.user_send_window = 4; evs.view_forget_timeout = PT24H; gcache.dir = /data/mysql/; gcache.freeze_purge_at_seqno = -1; gcache.keep_pages_count = 0; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = galera.cache; gcache.page_size = 128M; gcache.recover = yes; gcache.size = 2G; gcomm.thread_prio = ; gcs.fc_debug = 0; gcs.fc_factor = 1.0; gcs.fc_limit = 100; gcs.fc_master_slave = no; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = no; gmcast.segment = 0; gmcast.version = 0; pc.announce_timeout = PT3S; pc.checksum = false; pc.ignore_quorum = false; pc.ignore_sb = false; pc.npvo = false; pc.recovery = true; pc.version = 0; pc.wait_prim = true; pc.wait_prim_timeout = PT30S; pc.weight = 1; protonet.backend = asio; protonet.version = 0; repl.causal_read_timeout = PT30S; repl.commit_order = 3; repl.key_format = FLAT8; repl.max_ws_size = 2147483647; repl.proto_max = 10; socket.checksum = 2; socket.recv_buf_size = 212992; socket.ssl_ca = ca.pem; socket.ssl_cert = server-cert.pem; socket.ssl_cipher = ; socket.ssl_compression = YES; socket.ssl_key = server-key.pem; 
      2020-05-12T16:54:40.852007Z 0 [Note] [MY-000000] [WSREP] Starting replication
      2020-05-12T16:54:40.852102Z 0 [Note] [MY-000000] [Galera] Connecting with bootstrap option: 1
      2020-05-12T16:54:40.852155Z 0 [Note] [MY-000000] [Galera] Setting GCS initial position to 00000000-0000-0000-0000-000000000000:-1
      2020-05-12T16:54:40.852250Z 0 [Note] [MY-000000] [Galera] protonet asio version 0
      2020-05-12T16:54:40.852396Z 0 [Note] [MY-000000] [Galera] Using CRC-32C for message checksums.
      2020-05-12T16:54:40.852491Z 0 [Note] [MY-000000] [Galera] initializing ssl context
      2020-05-12T16:54:40.852665Z 0 [Note] [MY-000000] [Galera] backend: asio
      2020-05-12T16:54:40.852800Z 0 [Note] [MY-000000] [Galera] gcomm thread scheduling priority set to other:0 
      2020-05-12T16:54:40.852929Z 0 [Note] [MY-000000] [Galera] Restoring primary-component from disk successful
      2020-05-12T16:54:40.871255Z 0 [Note] [MY-000000] [Galera] GMCast version 0
      2020-05-12T16:54:40.871397Z 0 [Note] [MY-000000] [Galera] (ee952e10, 'ssl://0.0.0.0:4567') listening at ssl://0.0.0.0:4567
      2020-05-12T16:54:40.871471Z 0 [Note] [MY-000000] [Galera] (ee952e10, 'ssl://0.0.0.0:4567') multicast: , ttl: 1
      2020-05-12T16:54:40.871725Z 0 [Note] [MY-000000] [Galera] EVS version 1
      2020-05-12T16:54:40.871849Z 0 [Note] [MY-000000] [Galera] gcomm: bootstrapping new group 'xtradb'
      2020-05-12T16:54:40.871935Z 0 [Note] [MY-000000] [Galera] start_prim is enabled, turn off pc_recovery
      2020-05-12T16:54:40.872201Z 0 [Note] [MY-000000] [Galera] EVS version upgrade 0 -> 1
      2020-05-12T16:54:40.872285Z 0 [Note] [MY-000000] [Galera] PC protocol upgrade 0 -> 1
      2020-05-12T16:54:40.872376Z 0 [Note] [MY-000000] [Galera] Node ee952e10 state primary
      2020-05-12T16:54:40.872476Z 0 [Note] [MY-000000] [Galera] Current view of cluster as seen by this node
      view (view_id(PRIM,ee952e10,9)
      memb {
       ee952e10,0
       }
      joined {
       }
      left {
       }
      partitioned {
       }
      )
      2020-05-12T16:54:40.872555Z 0 [Note] [MY-000000] [Galera] Save the discovered primary-component to disk
      2020-05-12T16:54:40.885288Z 0 [Note] [MY-000000] [Galera] clear restored view
      2020-05-12T16:54:40.885443Z 0 [Note] [MY-000000] [Galera] gcomm: connected
      2020-05-12T16:54:40.885636Z 0 [Note] [MY-000000] [Galera] Changing maximum packet size to 64500, resulting msg size: 32636
      2020-05-12T16:54:40.885871Z 0 [Note] [MY-000000] [Galera] Shifting CLOSED -> OPEN (TO: 0)
      2020-05-12T16:54:40.886047Z 0 [Note] [MY-000000] [Galera] Opened channel 'xtradb'
      2020-05-12T16:54:40.886237Z 0 [Note] [MY-000000] [Galera] New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 1
      2020-05-12T16:54:40.886449Z 1 [Note] [MY-000000] [WSREP] Starting rollbacker thread 1
      2020-05-12T16:54:40.886592Z 0 [Note] [MY-000000] [Galera] Starting new group from scratch: 45a4c967-9471-11ea-a66d-130a23cd9c77
      2020-05-12T16:54:40.886710Z 2 [Note] [MY-000000] [WSREP] Starting applier thread 2
      2020-05-12T16:54:40.886877Z 0 [Note] [MY-000000] [Galera] STATE_EXCHANGE: sent state UUID: 45a4d51f-9471-11ea-a656-a3d6596e941c
      2020-05-12T16:54:40.887027Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: sent state msg: 45a4d51f-9471-11ea-a656-a3d6596e941c
      2020-05-12T16:54:40.887160Z 0 [Note] [MY-000000] [Galera] STATE EXCHANGE: got state msg: 45a4d51f-9471-11ea-a656-a3d6596e941c from 0 (node01)
      2020-05-12T16:54:40.887348Z 0 [Note] [MY-000000] [Galera] Quorum results:
       version = 6,
       component = PRIMARY,
       conf_id = 0,
       members = 1/1 (primary/total),
       act_id = 0,
       last_appl. = 0,
       protocols = 2/10/4 (gcs/repl/appl),
       vote policy= 0,
       group UUID = 45a4c967-9471-11ea-a66d-130a23cd9c77
      2020-05-12T16:54:40.887502Z 0 [Note] [MY-000000] [Galera] Flow-control interval: [100, 100]
      2020-05-12T16:54:40.887660Z 0 [Note] [MY-000000] [Galera] Restored state OPEN -> JOINED (1)
      2020-05-12T16:54:40.887802Z 0 [Note] [MY-000000] [Galera] Member 0.0 (node01) synced with group.
      2020-05-12T16:54:40.887969Z 2 [Note] [MY-000000] [Galera] ####### processing CC 1, local, ordered
      2020-05-12T16:54:40.888189Z 0 [Note] [MY-000000] [Galera] Shifting JOINED -> SYNCED (TO: 1)
      2020-05-12T16:54:40.888321Z 2 [Note] [MY-000000] [Galera] Drain monitors from -1 upto -1
      2020-05-12T16:54:40.888541Z 2 [Note] [MY-000000] [Galera] REPL Protocols: 10 (5, 3)
      2020-05-12T16:54:40.888771Z 2 [Note] [MY-000000] [Galera] ####### My UUID: ee952e10-946f-11ea-b21c-3205d47b4024
      2020-05-12T16:54:40.888997Z 2 [Note] [MY-000000] [Galera] Server node01 connected to cluster at position 45a4c967-9471-11ea-a66d-130a23cd9c77:1 with ID ee952e10-946f-11ea-b21c-3205d47b4024
      2020-05-12T16:54:40.889227Z 2 [Note] [MY-000000] [WSREP] Server status change disconnected -> connected
      [wsrep_notify] called with args : --status connected
      [wsrep_notify] Status Update xtradb-bootstrap-57ffc49947-c96v4 status=connected
      2020-05-12T16:54:40.898326Z 2 [Note] [MY-000000] [Galera] ####### ST not required
      2020-05-12T16:54:40.898565Z 2 [Note] [MY-000000] [Galera] Cert index reset to 00000000-0000-0000-0000-000000000000:-1 (proto: 10), state transfer needed: no
      2020-05-12T16:54:40.898832Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
      2020-05-12T16:54:40.899149Z 2 [Note] [MY-000000] [Galera] ####### Assign initial position for certification: 00000000-0000-0000-0000-000000000000:-1, protocol version: 5
      2020-05-12T16:54:40.899363Z 2 [Note] [MY-000000] [Galera] ####### Adjusting cert position: -1 -> 1
      2020-05-12T16:54:40.899570Z 0 [Note] [MY-000000] [Galera] Service thread queue flushed.
      2020-05-12T16:54:40.899868Z 2 [Note] [MY-000000] [Galera] ####### Setting monitor position to 1
      2020-05-12T16:54:40.915639Z 2 [Note] [MY-000000] [Galera] ####### Resetting gcache as cluster is being bootstrapped
      2020-05-12T16:54:40.915837Z 2 [Note] [MY-000000] [Galera] GCache history reset: old(db204094-9470-11ea-9032-4277165867e8:1 -> 45a4c967-9471-11ea-a66d-130a23cd9c77:0
      2020-05-12T16:54:43.003023Z 2 [Note] [MY-000000] [Galera] Flushing memory map to disk...
      2020-05-12T16:54:45.378325Z 2 [Note] [MY-000000] [Galera] GCache DEBUG: RingBuffer::seqno_reset(): discarded 168 bytes
      2020-05-12T16:54:45.378509Z 2 [Note] [MY-000000] [Galera] GCache DEBUG: RingBuffer::seqno_reset(): found 0/1 locked buffers
      2020-05-12T16:54:45.388775Z 2 [Note] [MY-000000] [Galera] Recording CC from group: 1
      2020-05-12T16:54:45.388951Z 2 [Note] [MY-000000] [Galera] Lowest cert index boundary for CC from group: 1
      2020-05-12T16:54:45.389113Z 2 [Note] [MY-000000] [Galera] Min available from gcache for CC from group: 1
      2020-05-12T16:54:45.389255Z 2 [Note] [MY-000000] [Galera] ================================================
      View:
       id: 45a4c967-9471-11ea-a66d-130a23cd9c77:1
       status: primary
       protocol_version: 4
       capabilities: MULTI-MASTER, CERTIFICATION, PARALLEL_APPLYING, REPLAY, ISOLATION, PAUSE, CAUSAL_READ, INCREMENTAL_WS, UNORDERED, PREORDERED, STREAMING, NBO
       final: no
       own_index: 0
       members(1):
       0: ee952e10-946f-11ea-b21c-3205d47b4024, node01
      =================================================
      2020-05-12T16:54:45.389383Z 2 [Note] [MY-000000] [WSREP] Server status change connected -> joiner
      [wsrep_notify] called with args : --status joiner
      [wsrep_notify] Status Update xtradb-bootstrap-57ffc49947-c96v4 status=joiner
      2020-05-12T16:54:45.395452Z 2 [Note] [MY-000000] [WSREP] Server status change joiner -> initializing
      [wsrep_notify] called with args : --status initializing
      [wsrep_notify] Status Update xtradb-bootstrap-57ffc49947-c96v4 status=initializing
      2020-05-12T16:54:45.405690Z 0 [Warning] [MY-013245] [Server] The SSL library function CRYPTO_set_mem_functions failed. This is typically caused by the SSL library already being used. As a result the SSL memory allocation will not be instrumented.
      2020-05-12T16:54:45.586878Z 3 [Note] [MY-000000] [WSREP] Initiating SST cancellation
      16:54:45 UTC - mysqld got signal 11 ;
      Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
      Thread pointer: 0x55a758bb7260
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 7f0bb51cdce0 thread_stack 0x46000
      /usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x41) [0x55a7566fd161]
      /usr/sbin/mysqld(handle_fatal_signal+0x323) [0x55a755764e93]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x153c0) [0x7f0c4be413c0]
      /usr/sbin/mysqld(log_online_follow_redo_log_one_pass()+0xec) [0x55a756899b0c]
      /usr/sbin/mysqld(srv_start(bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x30ae) [0x55a7569b326e]
      /usr/sbin/mysqld(+0x2336deb) [0x55a7567e9deb]
      /usr/sbin/mysqld(dd::bootstrap::DDSE_dict_init(THD*, dict_init_mode_t, unsigned int)+0x9a) [0x55a756495aaa]
      /usr/sbin/mysqld(dd::upgrade_57::do_pre_checks_and_initialize_dd(THD*)+0x3fc) [0x55a7566c761c]
      /usr/sbin/mysqld(+0x13bed18) [0x55a755871d18]
      /usr/sbin/mysqld(+0x2785eba) [0x55a756c38eba]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609) [0x7f0c4be35609]
      /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7f0c4b611103]
      Trying to get some variables.
      Some pointers may be invalid and cause the dump to abort.
      Query (0): Connection ID (thread ID): 3
      Status: NOT_KILLED
      You may download the Percona XtraDB Cluster operations manual by visiting
      http://www.percona.com/software/percona-xtradb-cluster/. You may find information
      in the manual which will help you identify the cause of the crash.
      

       

        Smart Checklist

          Attachments

            Issue Links

              Activity

                People

                Assignee:
                Unassigned
                Reporter:
                cameronbraid Cameron Braid
                Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                  Dates

                  Created:
                  Updated: