Uploaded image for project: 'Percona Server for MySQL'
  1. Percona Server for MySQL
  2. PS-4822

Migration from Percona-XtraDB-Cluster-server-56-5.6.26->Percona-Server-server-56-5.6.39

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 5.6.39-83.1
    • Fix Version/s: 5.6.41-84.1
    • Component/s: None
    • Labels:
      None

      Description

      Hi, during migration from Percona-XtraDB-Cluster 5.6 to Percona-Server-56-5.6.39 I stepped on the issue described below, it was reproduced on 2 different MySQL servers.

      After upgrade MySQL didn't start, here is stack trace.

      #0  __pthread_kill (threadid=<value optimized out>, signo=<value optimized out>) at ../nptl/sysdeps/unix/sysv/linux/pthread_kill.c:63
      #1  0x000000000065ba1d in handle_fatal_signal (sig=6) at /usr/src/debug/percona-server-5.6.39-83.1/sql/signal_handler.cc:236
      #2  <signal handler called>
      #3  0x00007f3355d8e4f5 in __bsd_signal (sig=28929, handler=0x7101) at ../sysdeps/posix/signal.c:34
      #4  0x00007f3355d8fcd5 in abort () at abort.c:118
      #5  0x0000000000a02a09 in pars_retrieve_table_def (node=0x34ca858, cursor_sym=0x0, search_cond=0x34cab20) at /usr/src/debug/percona-server-5.6.39-83.1/storage/innobase/pars/pars0pars.cc:865
      #6  pars_update_statement (node=0x34ca858, cursor_sym=0x0, search_cond=0x34cab20) at /usr/src/debug/percona-server-5.6.39-83.1/storage/innobase/pars/pars0pars.cc:1281
      #7  0x0000000000b3c80b in yyparse () at pars0grm.y:442
      #8  0x0000000000a04311 in pars_sql (info=0x34d8918, str=0xc4cd10 "PROCEDURE P () IS\nBEGIN\n  DELETE FROM SYS_ZIP_DICT_COLS\n    WHERE TABLE_ID = :table_id;\nEND;\n") at /usr/src/debug/percona-server-5.6.39-83.1/storage/innobase/pars/pars0pars.cc:2244
      #9  0x0000000000a082f6 in que_eval_sql (info=0x34d8918, sql=0xc4cd10 "PROCEDURE P () IS\nBEGIN\n  DELETE FROM SYS_ZIP_DICT_COLS\n    WHERE TABLE_ID = :table_id;\nEND;\n", reserve_dict_mutex=0, trx=0x34600d8) at /usr/src/debug/percona-server-5.6.39-83.1/storage/innobase/que/que0que.cc:1260
      Unhandled dwarf expression opcode 0xf3#10 0x0000000000a2faf9 in row_drop_table_for_mysql (name=
      ) at /usr/src/debug/percona-server-5.6.39-83.1/storage/innobase/row/row0mysql.cc:4833
      #11 0x0000000000a3168d in row_mysql_drop_temp_tables () at /usr/src/debug/percona-server-5.6.39-83.1/storage/innobase/row/row0mysql.cc:5086
      #12 0x00000000009dc19e in recv_recovery_rollback_active () at /usr/src/debug/percona-server-5.6.39-83.1/storage/innobase/log/log0recv.cc:3576
      #13 0x0000000000a599d3 in innobase_start_or_create_for_mysql () at /usr/src/debug/percona-server-5.6.39-83.1/storage/innobase/srv/srv0start.cc:2647
      Unhandled dwarf expression opcode 0xf3#14 0x0000000000999e62 in innobase_init (p=
      ) at /usr/src/debug/percona-server-5.6.39-83.1/storage/innobase/handler/ha_innodb.cc:4026
      #15 0x000000000059c5c8 in ha_initialize_handlerton (plugin=0x3344938) at /usr/src/debug/percona-server-5.6.39-83.1/sql/handler.cc:696
      #16 0x00000000006eab41 in plugin_initialize (plugin=0x3344938) at /usr/src/debug/percona-server-5.6.39-83.1/sql/sql_plugin.cc:1166
      Unhandled dwarf expression opcode 0xf3#17 0x00000000006f1112 in plugin_init (argc=
      ) at /usr/src/debug/percona-server-5.6.39-83.1/sql/sql_plugin.cc:1463
      #18 0x0000000000594e6d in init_server_components () at /usr/src/debug/percona-server-5.6.39-83.1/sql/mysqld.cc:5230
      #19 0x0000000000595eed in mysqld_main (argc=77, argv=0x3233e58) at /usr/src/debug/percona-server-5.6.39-83.1/sql/mysqld.cc:5874
      #20 0x00007f3355d7ad20 in __libc_start_main (main=0x570b80 <main(int, char**)>, argc=9, ubp_av=0x7ffe8d573dc8, init=<value optimized out>, fini=<value optimized out>, rtld_fini=<value optimized out>, stack_end=0x7ffe8d573db8) at libc-start.c:258
      #21 0x00000000005877fd in _start ()
       
       
      Here is root cause details.
       
      During InnoDB initialization (innobase_init func in /usr/src/debug/percona-server-5.6.39-83.1/storage/innobase/handler/ha_innodb.cc)  it was cleaning up some temp tables that ??were not cleaned properly during stop??. During this procedure MySQL tried to clean up everything gracefully.... 
      It was trying to cleanup SYS_ZIP_DICT_COLS table that related to plugin(XTRADB_ZIP_DICT/XTRADB_ZIP_DICT_COLS see https://www.percona.com/doc/percona-server/LATEST/flexibility/compressed_columns.html)  that was introduced with Percona-server 5.6.39. 
       
      So what happened: because this plugin was not initialized, this table(SYS_ZIP_DICT_COLS) didn't exist and MySQL tried to remove record from SYS_ZIP_DICT_COLS table that related to temp/#sql1966_14_17d.(see below)
       
      /usr/src/debug/percona-server-5.6.39-83.1/storage/innobase/pars/pars0pars.cc:
       
                      sym_node->table = dict_table_open_on_name(
                              sym_node->name, TRUE, FALSE, DICT_ERR_IGNORE_NONE);
      //
      // sym_node STRUCTURE after opening:
      // {common = {type = 16, parent = 0x0, brother = 0x0, val = {data = 0x0, ext = 0, len = 4294967295, type = {prtype = 0, mtype = 0, len = 0, mbminmaxlen = 0}}, val_buf_size = 0}, indirection = 0x0, alias = 0x0, col_var_list = {prev = 0x0, next = 0x0}, // // copy_val = 0, field_nos = {0, 0}, resolved = 1, token_type = SYM_TABLE_REF_COUNTED, name = 0x34ca840 "SYS_ZIP_DICT_COLS", name_len = 17, table = 0x0, col_no = 0, prefetch_buf = 0x0, cursor_def = 0x0, param_type = 0, sym_table = 0x34ca5c8, // // sym_list = {prev = 0x34ca688, next = 0x34ca960}, like_node = 0x0}
      (gdb) p *sym_node->table
      Cannot access memory at address 0x0
       
                      ut_a(sym_node->table != NULL); <-BackTrace happened, probably due address 0x0
       ** 

      #10 0x0000000000a2faf9 in row_drop_table_for_mysql (name=) at /usr/src/debug/percona-server-5.6.39-83.1/storage/innobase/row/row0mysql.cc:4833
              is_temp = 1
              err = DB_SUCCESS
              space_id = 0
              tablename = 0x34c4428 "tmp/#sql1966_14_17d" <-TEMP table that was handeled.

        Smart Checklist

          Attachments

            Activity

              People

              Assignee:
              Unassigned
              Reporter:
              tsolodov Timur Solodovnikov
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: