Uploaded image for project: 'Percona Server for MySQL'
  1. Percona Server for MySQL
  2. PS-6876

MySQL 8 Crash with MASTER_COMPRESSION_ALGORITHMS / slave_compressed_protocol

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Priority: Medium
    • Resolution: Fixed
    • Affects Version/s: 8.0.18-9
    • Fix Version/s: 8.0.20-11
    • Component/s: None
    • Labels:
      None

      Description

      MySQL will crash with below stack trace if the slave fails to connect to master and then later succeed.

       

      23:33:15 UTC - mysqld got signal 11 ;
       Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
       Thread pointer: 0x4536510
       Attempting backtrace. You can use the following information to find out
       where mysqld died. If you see no messages after this, something went
       terribly wrong...
       stack_bottom = 7fa4356d54b8 thread_stack 0x46000
       /usr/sbin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x3d) [0x1fb737d]
       /usr/sbin/mysqld(handle_fatal_signal+0x333) [0xfe0e23]
       /lib64/libpthread.so.0(+0xf5d0) [0x7fa44c6765d0]
       /usr/sbin/mysqld(mysql_compress_context_init(mysql_compress_context*, enum_compression_algorithm, unsigned int)+0) [0x1faf330]
       /usr/sbin/mysqld() [0xfc3645]
       /usr/sbin/mysqld(mysql_real_connect+0x99) [0xfc6f29]
       /usr/sbin/mysqld() [0xfc8a49]
       /usr/sbin/mysqld() [0x1c5039b]
       /usr/sbin/mysqld() [0x1c510a5]
       /usr/sbin/mysqld(handle_slave_io+0xbb1) [0x1c5ba11]
       /usr/sbin/mysqld() [0x24be55c]
       /lib64/libpthread.so.0(+0x7dd5) [0x7fa44c66edd5]
       /lib64/libc.so.6(clone+0x6d) [0x7fa44a7f602d]
      Trying to get some variables.
       Some pointers may be invalid and cause the dump to abort.
       Query (0): Connection ID (thread ID): 6
       Status: NOT_KILLED
      The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
       information that should help you find out what is causing the crash.
      

       

      GDB stack trace:

       

      #0 mysql_compress_context_init (cmp_ctx=0x40, algorithm=algorithm@entry=MYSQL_ZLIB, compression_level=3) at /export/home/pb2/build/sb_0-37170812-1575922551.75/rpm/BUILD/mysql-8.0.19/mysql-8.0.19/mysys/my_compress.cc:61
       #1 0x0000000000fc3645 in csm_prep_select_database (ctx=<optimized out>) at /export/home/pb2/build/sb_0-37170812-1575922551.75/rpm/BUILD/mysql-8.0.19/mysql-8.0.19/sql-common/client.cc:6457
       #2 0x0000000000fc6f29 in mysql_real_connect (mysql=mysql@entry=0x7fa1080099f0, host=host@entry=0x58f2be4 "192.168.54.120", user=user@entry=0x58f2ce5 "repl", passwd=passwd@entry=0x7fa15a40f3c0 "sekret", db=db@entry=0x0, port=<optimized out>, unix_socket=0x0, client_flag=2147483648) at /export/home/pb2/build/sb_0-37170812-1575922551.75/rpm/BUILD/mysql-8.0.19/mysql-8.0.19/sql-common/client.cc:5651
       #3 0x0000000001c506f9 in connect_to_master(THD*, MYSQL*, Master_info*, bool, bool) () at /export/home/pb2/build/sb_0-37170812-1575922551.75/rpm/BUILD/mysql-8.0.19/mysql-8.0.19/include/mysql/psi/mysql_cond.h:241
       #4 0x0000000001c5b0b6 in safe_connect (mi=0x58f2480, mysql=0x7fa1080099f0, thd=<optimized out>) at /export/home/pb2/build/sb_0-37170812-1575922551.75/rpm/BUILD/mysql-8.0.19/mysql-8.0.19/sql/rpl_slave.cc:7930
       #5 handle_slave_io () at /export/home/pb2/build/sb_0-37170812-1575922551.75/rpm/BUILD/mysql-8.0.19/mysql-8.0.19/sql/rpl_slave.cc:5246
       #6 0x00000000024be55c in pfs_spawn_thread (arg=0x5ac8970) at /export/home/pb2/build/sb_0-37170812-1575922551.75/rpm/BUILD/mysql-8.0.19/mysql-8.0.19/storage/perfschema/pfs.cc:2854
       #7 0x00007fa1713a8dd5 in start_thread () from /lib64/libpthread.so.0
       #8 0x00007fa16f53002d in clone () from /lib64/libc.so.6
      (gdb) p cmp_ctx
       +p cmp_ctx
       $1 = (mysql_compress_context *) 0x40
       (gdb) p *cmp_ctx
       +p *cmp_ctx
       Cannot access memory at address 0x40
       (gdb) frame 1
       +frame 1
       #1 0x0000000000fc3645 in csm_prep_select_database (ctx=<optimized out>) at /export/home/pb2/build/sb_0-37170812-1575922551.75/rpm/BUILD/mysql-8.0.19/mysql-8.0.19/sql-common/client.cc:6457
       6457 mysql_compress_context_init(&server_ext->compress_ctx, algorithm,
       (gdb) list
       +list
       6452 server_ext =
       6453 static_cast<NET_SERVER *>(MYSQL_EXTENSION_PTR(mysql)->server_extn);
       6454 net->extension = server_ext;
       6455 }
       6456 DBUG_ASSERT(server_ext != nullptr);
       6457 mysql_compress_context_init(&server_ext->compress_ctx, algorithm,
       6458 compress_level);
       6459 #endif
       6460 }
       6461 #ifdef CHECK_LICENSE
      

       

      Seems like as part of the connection close of the first attempt, we have erased (MYSQL_EXTENSION_PTR(mysql)->server_extn).

      How to repeat:

      • Start a master slave replication
      • On slave, execute:
      STOP SLAVE;
       CHANGE MASTER TO MASTER_COMPRESSION_ALGORITHMS='zlib';
       START SLAVE;
      • Once slave has connected, stop your master and wait until the slave disconnects, fail to connect again and enter the loop where it keeps trying to reconnect:
      2020-02-20T23:59:08.231116Z 4 [ERROR] [MY-010584] [Repl] Slave I/O for channel '': error connecting to master 'repl@192.168.54.120:3306' - retry-time: 15 retries: 1 message: Can't connect to MySQL server on '192.168.54.120' (111), Error_code: MY-002003
      • Start your master again. Slave will crash as soon as it connects.

       

      Workaround:

      Disable slave compression.

        Smart Checklist

          Attachments

            Issue Links

              Activity

                People

                Assignee:
                Unassigned
                Reporter:
                marcelo.altmann Marcelo Altmann
                Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                  Dates

                  Created:
                  Updated:
                  Resolved: