Uploaded image for project: 'Percona XtraBackup'
  1. Percona XtraBackup
  2. PXB-1894

PXB crashes during incremental backup prepare for tablespace encryption

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Priority: Medium
    • Resolution: Fixed
    • Affects Version/s: 2.4.14
    • Fix Version/s: 8.0.7, 2.4.16
    • Component/s: None
    • Labels:
      None

      Description

      Initialize and start PS 5.7.26-29 with tablespace encryption enabled(keyring/keyring_vault)

      /home/mchawla/PS040619_5_7_26_29_debug/bin/mysqld --no-defaults --core-file --basedir=/home/mchawla/PS040619_5_7_26_29_debug --tmpdir=/home/mchawla/PS040619_5_7_26_29_debug/data --datadir=/home/mchawla/PS040619_5_7_26_29_debug/data --plugin-load-add=tokudb=ha_tokudb.so --tokudb-check-jemalloc=0 --plugin-load-add=rocksdb=ha_rocksdb.so --socket=/home/mchawla/PS040619_5_7_26_29_debug/socket.sock --port=11331 --log-error=/home/mchawla/PS040619_5_7_26_29_debug/log/master.err --server-id=100 --log-bin=binlog --early-plugin-load=keyring_vault=keyring_vault.so --keyring_vault_config=/home/mchawla/test_mode/vault/keyring_vault.cnf --innodb_encrypt_tables=ON --innodb_encrypt_online_alter_logs=ON --innodb_temp_tablespace_encrypt=ON --log-slave-updates --gtid-mode=ON --enforce-gtid-consistency --binlog-format=row --master_verify_checksum=ON --binlog_checksum=CRC32 --encrypt-tmp-files

      Creating innodb data in database with encrypted tables

      num_tables=10
       for ((i=1; i<=${num_tables}; i++)); do
       echo "Creating the table sbtest$i..."
       ${mysqldir}/bin/mysql -uroot -S${mysqldir}/socket.sock -e "CREATE TABLE test.sbtest$i (id int(11) NOT NULL AUTO_INCREMENT, k int(11) NOT NULL DEFAULT '0', c char(120) NOT NULL DEFAULT '', pad char(60) NOT NULL DEFAULT '', PRIMARY KEY (id), KEY k_1 (k)) ENGINE=InnoDB DEFAULT CHARSET=latin1 ENCRYPTION='Y';"
       done
      
      echo "Adding data in tables..."
      sysbench /usr/share/sysbench/oltp_insert.lua --tables=${num_tables} --mysql-db=test --mysql-user=root --threads=50 --db-driver=mysql --mysql-socket=${mysqldir}/socket.sock --time=30 run >/dev/null 2>&1

      Taking full backup

      /home/mchawla/pxb_2_4_14_release_debug/bin/xtrabackup --user=root --password='' --backup --target-dir=/home/mchawla/dbbackup_08_07_2019/full -S /home/mchawla/PS040619_5_7_26_29_debug/socket.sock --datadir=/home/mchawla/PS040619_5_7_26_29_debug/data --keyring_vault_config=/home/mchawla/test_mode/vault/keyring_vault.cnf --xtrabackup-plugin-dir=/home/mchawla/pxb_2_4_14_release_debug/bin/../lib/plugin

      Adding data in database

      sysbench /usr/share/sysbench/oltp_insert.lua --tables=${num_tables} --mysql-db=test --mysql-user=root --threads=50 --db-driver=mysql --mysql-socket=${mysqldir}/socket.sock --time=20 run >/dev/null 2>&1 &
      sleep 5
      

      Taking incremental backup

      /home/mchawla/pxb_2_4_14_release_debug/bin/xtrabackup --user=root --password='' --backup --target-dir=/home/mchawla/dbbackup_08_07_2019/inc --incremental-basedir=/home/mchawla/dbbackup_08_07_2019/full -S /home/mchawla/PS040619_5_7_26_29_debug/socket.sock --datadir=/home/mchawla/PS040619_5_7_26_29_debug/data --keyring_vault_config=/home/mchawla/test_mode/vault/keyring_vault.cnf --xtrabackup-plugin-dir=/home/mchawla/pxb_2_4_14_release_debug/bin/../lib/plugin

      Preparing full backup

      /home/mchawla/pxb_2_4_14_release_debug/bin/xtrabackup --user=root --password='' --prepare --apply-log-only --target_dir=/home/mchawla/dbbackup_08_07_2019/full --keyring_vault_config=/home/mchawla/test_mode/vault/keyring_vault.cnf --xtrabackup-plugin-dir=/home/mchawla/pxb_2_4_14_release_debug/bin/../lib/plugin

      Preparing incremental backup

      /home/mchawla/pxb_2_4_14_release_debug/bin/xtrabackup --user=root --password='' --prepare --target_dir=/home/mchawla/dbbackup_08_07_2019/full --incremental-dir=/home/mchawla/dbbackup_08_07_2019/inc --keyring_vault_config=/home/mchawla/test_mode/vault/keyring_vault.cnf --xtrabackup-plugin-dir=/home/mchawla/pxb_2_4_14_release_debug/bin/../lib/plugin

      PXB crashes

      InnoDB: Progress in percent: 72 73 74 75 76 2019-07-05 10:08:01 0x7f1ae98fd880 InnoDB: Assertion failure in thread 139753564395648 in file log0recv.cc line 1899
      InnoDB: We intentionally generate a memory trap.
      InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
      InnoDB: If you get repeated assertion failures or crashes, even
      InnoDB: immediately after the mysqld startup, there may be
      InnoDB: corruption in the InnoDB tablespace. Please refer to
      InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
      InnoDB: about forcing recovery.
      10:08:01 UTC - xtrabackup got signal 6 ;
      This could be because you hit a bug or data is corrupted.
      This error can also be caused by malfunctioning hardware.
      Attempting to collect some information that could help diagnose the problem.
      As this is a crash and something is definitely wrong, the information
      collection process might fail.
      Thread pointer: 0x0
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0 thread_stack 0x10000
      /home/mchawla/pxb_2_4_15_debug/bin/xtrabackup(my_print_stacktrace+0x35)[0x1596eaf]
      /home/mchawla/pxb_2_4_15_debug/bin/xtrabackup(handle_fatal_signal+0x2a5)[0x133958e]
      /usr/lib64/libpthread.so.0(+0xf5d0)[0x7f1ae94ec5d0]
      /usr/lib64/libc.so.6(gsignal+0x37)[0x7f1ae7166207]
      /usr/lib64/libc.so.6(abort+0x148)[0x7f1ae71678f8]
      /home/mchawla/pxb_2_4_15_debug/bin/xtrabackup[0x116544d]
      /home/mchawla/pxb_2_4_15_debug/bin/xtrabackup[0x11a6e9f]
      /home/mchawla/pxb_2_4_15_debug/bin/xtrabackup(_Z22recv_recover_page_funcmP11buf_block_t+0x666)[0x11a91f3]
      /home/mchawla/pxb_2_4_15_debug/bin/xtrabackup(_Z20buf_page_io_completeP10buf_page_tb+0x82f)[0x11c52cb]
      /home/mchawla/pxb_2_4_15_debug/bin/xtrabackup[0x1136693]
      /home/mchawla/pxb_2_4_15_debug/bin/xtrabackup(_Z19buf_read_recv_pagesbmPKmm+0x281)[0x1137c5a]
      /home/mchawla/pxb_2_4_15_debug/bin/xtrabackup[0x11a95f6]
      /home/mchawla/pxb_2_4_15_debug/bin/xtrabackup(_Z26recv_apply_hashed_log_recsm+0x4e7)[0x11a9aeb]
      /home/mchawla/pxb_2_4_15_debug/bin/xtrabackup(_Z34innobase_start_or_create_for_mysqlv+0x2048)[0x1103962]
      /home/mchawla/pxb_2_4_15_debug/bin/xtrabackup[0xe00c5d]
      /home/mchawla/pxb_2_4_15_debug/bin/xtrabackup[0xe0ccb4]
      /home/mchawla/pxb_2_4_15_debug/bin/xtrabackup(main+0x9ef)[0xe0fa77]
      /usr/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f1ae71523d5]
      /home/mchawla/pxb_2_4_15_debug/bin/xtrabackup[0xdf9819]

      Notes:
      1. The issue comes when PS is initialized and the database is created. Subsequent incremental backups do not reproduce the issue.
      2. The issue was also reproduced in an earlier PXB 2.4.14 version.

        Smart Checklist

          Attachments

            Activity

              People

              Assignee:
              sergei.glushchenko Sergei Glushchenko (Inactive)
              Reporter:
              manish.chawla Manish Chawla
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - Not Specified
                  Not Specified
                  Logged:
                  Time Spent - 4 days, 1 hour, 15 minutes
                  4d 1h 15m