Uploaded image for project: 'Percona XtraBackup'
  1. Percona XtraBackup
  2. PXB-1954

PXB gets stuck in prepare of full backup for encrypted tablespace

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Priority: High
    • Resolution: Fixed
    • Affects Version/s: 8.0.8
    • Fix Version/s: 8.0.8
    • Component/s: None
    • Labels:
      None

      Description

      PS8.0.17-8 is running with encryption options

      /home/mchawla/PS241019_8_0_17_8_debug/bin/mysqld --no-defaults --core-file --basedir=/home/mchawla/PS241019_8_0_17_8_debug --tmpdir=/home/mchawla/PS241019_8_0_17_8_debug/data --datadir=/home/mchawla/PS241019_8_0_17_8_debu/data --plugin-load-add=tokudb=ha_tokudb.so --tokudb-check-jemalloc=0 --plugin-load-add=rocksdb=ha_rocksdb.so --socket=/home/mchawla/PS241019_8_0_17_8_debug/socket.sock --port=11764 --log-error=/home/mchawla/PS241019_8_0_17_8_debug/log/master.err --server-id=100 --log-bin=binlog --early-plugin-load=keyring_file.so --keyring_file_data=/home/mchawla/PS241019_8_0_17_8_debug/keyring --innodb-undo-log-encrypt --innodb-redo-log-encrypt --default-table-encryption=ON --innodb_encrypt_online_alter_logs=ON --innodb_temp_tablespace_encrypt=ON --log-slave-updates --gtid-mode=ON --enforce-gtid-consistency --binlog-format=row --master_verify_checksum=ON --binlog_checksum=CRC32 --encrypt-tmp-files --innodb_sys_tablespace_encrypt --innodb_parallel_dblwr_encrypt --binlog-rotate-encryption-master-key-at-startup --table-encryption-privilege-check=ON --innodb-default-encryption-key-id=4294967295 --innodb-encryption-threads=10

      Create innodb encrypted tables(10). Add some data.

      for ((i=1; i<=${num_tables}; i++)); do
       echo "Creating the table sbtest$i..."
       ${mysqldir}/bin/mysql -uroot -S${mysqldir}/socket.sock -e "CREATE TABLE test.sbtest$i (id int(11) NOT NULL AUTO_INCREMENT, k int(11) NOT NULL DEFAULT '0', c char(120) NOT NULL DEFAULT '', pad char(60) NOT NULL DEFAULT '', PRIMARY KEY (id), KEY k_1 (k)) ENGINE=InnoDB DEFAULT CHARSET=latin1 ENCRYPTION='Y';"
       done
      echo "Adding data in tables..."
       sysbench /usr/share/sysbench/oltp_insert.lua --tables=${num_tables} --mysql-db=test --mysql-user=root --threads=50 --db-driver=mysql --mysql-socket=${mysqldir}/socket.sock --time=30 run >/dev/null 2>&1
       fi

      Run the loop continuously

      add_drop_full_text_index() {
      
      # This function adds and drops a full text index in a table
      echo "Add and drop a full text index in the test.sbtest1 table"
       ( for ((i=1; i<=10; i++)); do
      
      Check if database is up otherwise exit the loop
       ${mysqldir}/bin//mysqladmin ping --user=root --socket=${mysqldir}/socket.sock 2>/dev/null 1>&2
       if [ "$?" -ne 0 ]; then
       break
       fi
       ${mysqldir}/bin/mysql -uroot -S${mysqldir}/socket.sock -e "CREATE FULLTEXT INDEX full_index on test.sbtest1 (pad);" >/dev/null 2>&1
       ${mysqldir}/bin/mysql -uroot -S${mysqldir}/socket.sock -e "DROP INDEX full_index on test.sbtest1;" >/dev/null 2>&1
       done ) &
       }

      Take full backup

      /home/mchawla/pxb_8_0_8_debug/bin/xtrabackup --user=root --password='' --backup --target-dir=/home/mchawla/dbbackup_06_11_2019/full -S /home/mchawla/PS241019_8_0_17_8_debug/socket.sock --datadir=/home/mchawla/PS241019_8_0_17_8_debug/data --keyring_file_data=/home/mchawla/PS241019_8_0_17_8_debug/keyring --xtrabackup-plugin-dir=/home/mchawla/pxb_8_0_8_debug/bin/../lib/plugin --lock-ddl

      Run a small load with sysbench

      # Innodb data
      sysbench /usr/share/sysbench/oltp_insert.lua --tables=${num_tables} --mysql-db=test --mysql-user=root --threads=50 --db-driver=mysql --mysql-socket=${mysqldir}/socket.sock --time=20 run >/dev/null 2>&1 &

      Take incremental backup

      /home/mchawla/pxb_8_0_8_debug/bin/xtrabackup --user=root --password='' --backup --target-dir=/home/mchawla/dbbackup_06_11_2019/inc --incremental-basedir=/home/mchawla/dbbackup_06_11_2019/full -S /home/mchawla/PS241019_8_0_17_8_debug/socket.sock --datadir=/home/mchawla/PS241019_8_0_17_8_debug/data --keyring_file_data=/home/mchawla/PS241019_8_0_17_8_debug/keyring --xtrabackup-plugin-dir=/home/mchawla/pxb_8_0_8_debug/bin/../lib/plugin --lock-ddl

      Prepare full backup

      /home/mchawla/pxb_8_0_8_debug/bin/xtrabackup --user=root --password='' --prepare --apply-log-only --target_dir=/home/mchawla/dbbackup_06_11_2019/full --keyring_file_data=/home/mchawla/PS241019_8_0_17_8_debug/keyring --xtrabackup-plugin-dir=/home/mchawla/pxb_8_0_8_debug/bin/../lib/plugin

      PXB gets stuck at this point.
      Logs attached: pxb_stuck_add_drop_full_text_index.tar.gz (Backup logs, thread_apply_all_bt_06_11.txt)

      Issue was observed for another scenario:

      change_row_format() {
          # This function changes the row format of a table
      
          echo "Change the row format of an innodb table"
          ( for ((i=1; i<=10; i++)); do
              # Check if database is up otherwise exit the loop
              ${mysqldir}/bin//mysqladmin ping --user=root --socket=${mysqldir}/socket.sock 2>/dev/null 1>&2
              if [ "$?" -ne 0 ]; then
                  break
              fi
              ${mysqldir}/bin/mysql -uroot -S${mysqldir}/socket.sock -e "ALTER TABLE test.sbtest2 ROW_FORMAT=COMPRESSED;" >/dev/null 2>&1
              ${mysqldir}/bin/mysql -uroot -S${mysqldir}/socket.sock -e "ALTER TABLE test.sbtest2 ROW_FORMAT=DYNAMIC;" >/dev/null 2>&1
              ${mysqldir}/bin/mysql -uroot -S${mysqldir}/socket.sock -e "ALTER TABLE test.sbtest2 ROW_FORMAT=COMPACT;" >/dev/null 2>&1
              ${mysqldir}/bin/mysql -uroot -S${mysqldir}/socket.sock -e "ALTER TABLE test.sbtest2 ROW_FORMAT=REDUNDANT;" >/dev/null 2>&1
          done ) &
      }

      Logs attached: pxb_stuck_row_format_change.tar.gz (Backup logs, thread_apply_all_bt.txt)

        Smart Checklist

          Attachments

            Activity

              People

              Assignee:
              sergei.glushchenko Sergei Glushchenko (Inactive)
              Reporter:
              manish.chawla Manish Chawla
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - Not Specified
                  Not Specified
                  Logged:
                  Time Spent - 3 hours, 30 minutes
                  3h 30m