Uploaded image for project: 'Percona Server for MySQL'
  1. Percona Server for MySQL
  2. PS-5909

Rotating Innodb Master key post upgrade crashes the server

    Details

    • Type: Bug
    • Status: Done
    • Priority: High
    • Resolution: Fixed
    • Affects Version/s: 8.0.16-7
    • Fix Version/s: 8.0.17-8
    • Component/s: None
    • Labels:
      None

      Description

      Steps to repro:

      1. create datadir on 5.7:

      ./bin/mysqld --no-defaults --datadir=`pwd`/data_57 --initialize-insecure --plugin-dir=/home/mohit.joshi/PS010819-percona-server-5.7.26-29-linux-x86_64-debug/lib/mysql/plugin/ --early-plugin-load=keyring_file.so --keyring_file_data=./data/mykey --innodb_sys_tablespace_encrypt=ON --innodb_undo_tablespaces=10

      2. start the server on 5.7:

      ./bin/mysqld --defaults-file=~/my.cnf --datadir=`pwd`/data_57 --port=21000 --socket=/tmp/mysql_21000.sock --lc-messages-dir=./share --plugin-dir=/home/mohit.joshi/PS010819-percona-server-5.7.26-29-linux-x86_64-debug/lib/mysql/plugin/ --early-plugin-load=keyring_file.so --keyring_file_data=./data/mykey --max-connections=1024 --log-error --general-log --log-error-verbosity=3 --core-file --innodb_undo_log_encrypt=ON --innodb_temp_tablespace_encrypt=ON --innodb_redo_log_encrypt=MASTER_KEY --innodb_encrypt_tables=ON --innodb_sys_tablespace_encrypt=ON --innodb_undo_tablespaces=5 --innodb_parallel_dblwr_encrypt=ON --innodb_encrypt_online_alter_logs=ON --encrypt_tmp_files=ON &

      3. Now run sysbench to run some Load:

      mysql -A -uroot -S/tmp/mysql_21000.sock -e "create user sysbench@'%' identified with mysql_native_password by 'test';"
      mysql -A -uroot -S/tmp/mysql_21000.sock -e "grant all on . to sysbench@'%';"
      mysql -A -uroot -S/tmp/mysql_21000.sock -e "drop database if exists sbtest;create database sbtest;"

      sysbench /usr/share/sysbench/oltp_insert.lua --mysql-db=sbtest --mysql-user=sysbench --mysql-password=test --db-driver=mysql --mysql-socket=/tmp/mysql_21000.sock --threads=1000 --tables=100 --table-size=1000 prepare

      sysbench /usr/share/sysbench/oltp_read_write.lua --mysql-db=sbtest --mysql-user=sysbench --mysql-password=test --db-driver=mysql --mysql-socket=/tmp/mysql_21000.sock --threads=100 --tables=100 --time=3000 --report-interval=1 --events=1870000000 --db-ps-mode=disable run &

      4. Shutdown the server from client

      5. Start the server on 8.0 using 5.7 datadir:

      ./bin/mysqld --defaults-file=~/my.cnf --datadir=/home/mohit.joshi/PS200819-percona-server-5.7.27-30-linux-x86_64-debug/data_57 --port=22000 --socket=/tmp/mysql_22000.sock --lc-messages-dir=./share --plugin-dir=/home/mohit.joshi/PS200819-percona-server-8.0.16-7-linux-x86_64-debug/lib/mysql/plugin/ --early-plugin-load=keyring_file.so --keyring_file_data=/home/mohit.joshi/PS200819-percona-server-5.7.27-30-linux-x86_64-debug/data_57/data/mykey --max-connections=1024 --log-error --general-log --log-error-verbosity=3 --core-file --innodb_undo_log_encrypt=ON --innodb_temp_tablespace_encrypt=ON --innodb_redo_log_encrypt=MASTER_KEY --default_table_encryption=ON --innodb_sys_tablespace_encrypt=ON --innodb_undo_tablespaces=2 --innodb_parallel_dblwr_encrypt=ON --innodb_encrypt_online_alter_logs=ON --encrypt_tmp_files=ON &

      6. Execute on upgraded server:
      ALTER INSTANCE ROTATE INNODB MASTER KEY;

       

      Stack trace:

      stack_bottom = 7f537c14dd18 thread_stack 0x46000
      /home/mohit.joshi/PS200819-percona-server-8.0.16-7-linux-x86_64-debug/bin/mysqld(my_print_stacktrace(unsigned char*, unsigned long)+0x55) [0x4808cca]
      /home/mohit.joshi/PS200819-percona-server-8.0.16-7-linux-x86_64-debug/bin/mysqld(handle_fatal_signal+0x2c6) [0x363a5b3]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390) [0x7f539bded390]
      /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x38) [0x7f539a0c5428]
      /lib/x86_64-linux-gnu/libc.so.6(abort+0x16a) [0x7f539a0c702a]
      /home/mohit.joshi/PS200819-percona-server-8.0.16-7-linux-x86_64-debug/bin/mysqld(ut_dbg_assertion_failed(char const*, char const*, unsigned long)+0x1aa) [0x4db6309]
      /home/mohit.joshi/PS200819-percona-server-8.0.16-7-linux-x86_64-debug/bin/mysqld(buf_page_get_gen(page_id_t const&, page_size_t const&, unsigned long, buf_block_t*, Page_fetch, char const*, unsigned long, mtr_t*, bool, dberr_t*)+0x2d8) [0x4e650b8]
      /home/mohit.joshi/PS200819-percona-server-8.0.16-7-linux-x86_64-debug/bin/mysqld(fsp_header_write_encryption(unsigned int, unsigned long, unsigned char*, bool, bool, mtr_t*)+0x8e) [0x4fb6be6]
      /home/mohit.joshi/PS200819-percona-server-8.0.16-7-linux-x86_64-debug/bin/mysqld(fsp_header_rotate_encryption(fil_space_t*, unsigned char*, mtr_t*)+0xf5) [0x4fb6efb]
      /home/mohit.joshi/PS200819-percona-server-8.0.16-7-linux-x86_64-debug/bin/mysqld(encryption_rotate_low(fil_space_t*)+0xeb) [0x4f78cbe]
      /home/mohit.joshi/PS200819-percona-server-8.0.16-7-linux-x86_64-debug/bin/mysqld(Fil_system::encryption_rotate_in_a_shard(Fil_shard*)+0x248) [0x4f79070]
      /home/mohit.joshi/PS200819-percona-server-8.0.16-7-linux-x86_64-debug/bin/mysqld(Fil_system::encryption_rotate_all()+0x80) [0x4f79184]
      /home/mohit.joshi/PS200819-percona-server-8.0.16-7-linux-x86_64-debug/bin/mysqld(fil_encryption_rotate()+0x13) [0x4f791d5]
      /home/mohit.joshi/PS200819-percona-server-8.0.16-7-linux-x86_64-debug/bin/mysqld(innobase_encryption_key_rotation()+0x136) [0x4a5b842]
      /home/mohit.joshi/PS200819-percona-server-8.0.16-7-linux-x86_64-debug/bin/mysqld(Rotate_innodb_master_key::execute()+0x228) [0x335bd4c]
      /home/mohit.joshi/PS200819-percona-server-8.0.16-7-linux-x86_64-debug/bin/mysqld(Sql_cmd_alter_instance::execute(THD*)+0x172) [0x3350e04]
      /home/mohit.joshi/PS200819-percona-server-8.0.16-7-linux-x86_64-debug/bin/mysqld(mysql_execute_command(THD*, bool)+0x61fe) [0x34487ba]
      /home/mohit.joshi/PS200819-percona-server-8.0.16-7-linux-x86_64-debug/bin/mysqld(mysql_parse(THD*, Parser_state*, bool)+0x6cd) [0x344b28b]
      /home/mohit.joshi/PS200819-percona-server-8.0.16-7-linux-x86_64-debug/bin/mysqld(dispatch_command(THD*, COM_DATA const*, enum_server_command)+0x1571) [0x343f7b4]
      /home/mohit.joshi/PS200819-percona-server-8.0.16-7-linux-x86_64-debug/bin/mysqld(do_command(THD*)+0x50b) [0x343dbcd]
      /home/mohit.joshi/PS200819-percona-server-8.0.16-7-linux-x86_64-debug/bin/mysqld() [0x3623dd1]
      /home/mohit.joshi/PS200819-percona-server-8.0.16-7-linux-x86_64-debug/bin/mysqld() [0x4902b76]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7f539bde36ba]
      /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f539a19741d]

      Trying to get some variables.
      Some pointers may be invalid and cause the dump to abort.
      Query (7f52c0097b38): alter instance rotate innodb master key

        Smart Checklist

          Attachments

            Activity

              People

              • Assignee:
                satya.bodapati Satya Bodapati
                Reporter:
                mohit.joshi Mohit Joshi
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - Not Specified
                  Not Specified
                  Logged:
                  Time Spent - 6 hours
                  6h