Uploaded image for project: 'Percona XtraBackup'
  1. Percona XtraBackup
  2. PXB-1903

PXB is stuck with redo log corruption when master key is rotated

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Priority: High
    • Resolution: Fixed
    • Affects Version/s: 8.0.6, 2.4.15
    • Fix Version/s: 8.0.7, 2.4.16
    • Component/s: None
    • Labels:
      None

      Description

      Initialize and start PS8.0.16-6(redo log is not encrypted)

      ./all_no_cl --early-plugin-load=keyring_file.so --keyring_file_data=$HOME/PS230719_8_0_16_6_debug/keyring --binlog-encryption --innodb-undo-log-encrypt --default-table-encryption=ON --innodb_encrypt_online_alter_logs=ON --innodb_temp_tablespace_encrypt=ON --log-bin=mysql-bin --log-slave-updates --gtid-mode=ON --enforce-gtid-consistency --binlog-format=row --master_verify_checksum=ON --binlog_checksum=CRC32 --encrypt-tmp-files --innodb_sys_tablespace_encrypt --innodb_parallel_dblwr_encrypt --binlog-rotate-encryption-master-key-at-startup --table-encryption-privilege-check=ON --innodb-default-encryption-key-id=4294967295 --innodb-encryption-threads=10

      Create data

      #!/bin/bash
      num_tables=10
      echo "Create an encrypted tablespace ts1"
      ${PWD}/bin/mysql -uroot -S${PWD}/socket.sock -e "CREATE TABLESPACE ts1 ADD DATAFILE 'ts1.ibd' ENCRYPTION = 'Y' Engine=InnoDB;"
      echo "Create an unencrypted tablespace ts1"
      ${PWD}/bin/mysql -uroot -S${PWD}/socket.sock -e "CREATE TABLESPACE ts2 ADD DATAFILE 'ts2.ibd' ENCRYPTION = 'N' Engine=InnoDB;"
      echo "Create tables in ts1"
      for ((i=1; i<=${num_tables}-5; i++)); do
       echo "Creating the table sbtest$i..."
       ${PWD}/bin/mysql -uroot -S${PWD}/socket.sock -e "CREATE TABLE test.sbtest$i (id int(11) NOT NULL AUTO_INCREMENT, k int(11) NOT NULL DEFAULT '0', c char(120) NOT NULL DEFAULT '', pad char(60) NOT NULL DEFAULT '', PRIMARY KEY (id), KEY k_1 (k)) ENGINE=InnoDB DEFAULT CHARSET=latin1 ENCRYPTION='Y' TABLESPACE ts1;"
      done
      echo "Create tables in ts2"
      for ((i=${num_tables}-4; i<=${num_tables}; i++)); do
       echo "Creating the table sbtest$i..."
       ${PWD}/bin/mysql -uroot -S${PWD}/socket.sock -e "CREATE TABLE test.sbtest$i (id int(11) NOT NULL AUTO_INCREMENT, k int(11) NOT NULL DEFAULT '0', c char(120) NOT NULL DEFAULT '', pad char(60) NOT NULL DEFAULT '', PRIMARY KEY (id), KEY k_1 (k)) ENGINE=InnoDB DEFAULT CHARSET=latin1 ENCRYPTION='N' TABLESPACE ts2;"
      done
      echo "Adding data in tables..."
      sysbench /usr/share/sysbench/oltp_insert.lua --tables=${num_tables} --mysql-db=test --mysql-user=root --threads=50 --db-driver=mysql --mysql-socket=${PWD}/socket.sock --time=30 run >/dev/null 2>&1

      Start a load using sysbench

      sysbench /usr/share/sysbench/oltp_insert.lua --tables=10 --mysql-db=test --mysql-user=root --threads=100 --db-driver=mysql --mysql-socket=${PWD}/socket.sock --time=200 --report-interval=5 run

      Take full backup and during the backup rotate the master key

      ./xtrabackup --user=root --backup --target-dir=$HOME/dbbackup_PS8/full -S $HOME/PS230719_8_0_16_6_debug/socket.sock --datadir=$HOME/PS230719_8_0_16_6_debug/data --keyring_file_data=$HOME/PS230719_8_0_16_6_debug/keyring --xtrabackup-plugin-dir=$HOME/pxb_8_0_7_debug/lib/plugin --transition-key=abc 2>&1 | tee full_backup_$(date +"%d_%m_%Y")_log
      ALTER INSTANCE ROTATE INNODB MASTER KEY;

      PXB displays the following logs continuously and keeps on running

      ############### CORRUPT LOG RECORD FOUND ###############
      Log record type 30, page 2:0. Log parsing proceeded successfully up to 29982004. Previous log record type 30, is multi 0 Recv offset 143148, prev 143148
      Hex dump starting 100 bytes before and ending 100 bytes after the corrupted record:
       len 200; hex 0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001f9e02002896006f6c43430000000336303432366566382d623338382d313165392d613734322d303032353930653962343438c538ca8c2b1d7329e298dad6181c4702bbbe502a00d7396578a746633ea86eb71d20bf9a3cfcb11612aac4eca2e7f8177189; asc ( olCC 60426ef8-b388-11e9-a742-002590e9b448 8 + s) G P* 9ex Fc> n < q ;
      Set innodb_force_recovery to ignore this error.
      Encryption information in the redo log of space 2 is invalid

      Note: The issue was reproduced with MS8.0.16 also. MS was started with the following options

      /home/mchawla/MS_8_0_16/bin/mysqld --no-defaults --core-file --basedir=/home/mchawla/MS_8_0_16 --tmpdir=/home/mchawla/MS_8_0_16/data --datadir=/home/mchawla/MS_8_0_16/data --socket=/home/mchawla/MS_8_0_16/socket.sock --port=17514 --log-error=/home/mchawla/MS_8_0_16/log/master.err --server-id=100 --early-plugin-load=keyring_file.so --keyring_file_data=/home/mchawla/MS_8_0_16/keyring --binlog-encryption --innodb-undo-log-encrypt --default-table-encryption=ON --log-bin=mysql-bin --log-slave-updates --gtid-mode=ON --enforce-gtid-consistency --binlog-format=row --master_verify_checksum=ON --binlog_checksum=CRC32 --binlog-rotate-encryption-master-key-at-startup --table-encryption-privilege-check=ON

      Full backup logs are attached.

        Smart Checklist

          Attachments

            Activity

              People

              Assignee:
              sergei.glushchenko Sergei Glushchenko (Inactive)
              Reporter:
              manish.chawla Manish Chawla
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - Not Specified
                  Not Specified
                  Logged:
                  Time Spent - 1 day, 4 hours, 26 minutes
                  1d 4h 26m