Uploaded image for project: 'Percona XtraBackup'
  1. Percona XtraBackup
  2. PXB-2097

PXB crashes in incremental backup prepare when row format is changed or full text index is created/dropped

Details

    • Bug
    • Status: Done
    • Medium
    • Resolution: Duplicate
    • 2.4.18
    • None
    • None
    • None
    • Yes

    Description

      Scenario 1: Row format is changed
      Start MS 5.7.29 with log-bin option
      Add data in PS using sysbench: 10 tables, 1000 rows in each table
      Change the row format of a table continuously

      ( for ((i=1; i<=10; i++)); do
       # Check if database is up otherwise exit the loop
       ${mysqldir}/bin//mysqladmin ping --user=root --socket=${mysqldir}/socket.sock 2>/dev/null 1>&2
       if [ "$?" -ne 0 ]; then
       break
       fi
       ${mysqldir}/bin/mysql -uroot -S${mysqldir}/socket.sock -e "ALTER TABLE test.sbtest2 ROW_FORMAT=COMPRESSED;" >/dev/null 2>&1
       ${mysqldir}/bin/mysql -uroot -S${mysqldir}/socket.sock -e "ALTER TABLE test.sbtest2 ROW_FORMAT=DYNAMIC;" >/dev/null 2>&1
       ${mysqldir}/bin/mysql -uroot -S${mysqldir}/socket.sock -e "ALTER TABLE test.sbtest2 ROW_FORMAT=COMPACT;" >/dev/null 2>&1
       ${mysqldir}/bin/mysql -uroot -S${mysqldir}/socket.sock -e "ALTER TABLE test.sbtest2 ROW_FORMAT=REDUNDANT;" >/dev/null 2>&1
       done ) &

      Take full backup with lock-ddl-per-table

      /home/mchawla/pxb_2_4_18_release_debug/bin/xtrabackup --user=root --password='' --backup --target-dir=/home/mchawla/dbbackup_16_01_2020/full -S /home/mchawla/MS_5_7_29/socket.sock --datadir=/home/mchawla/MS_5_7_29/data --lock-ddl-per-table

      Start a data load on the server using sysbench
      Take incremental backup with lock-ddl-per-table

      /home/mchawla/pxb_2_4_18_release_debug/bin/xtrabackup --user=root --password='' --backup --target-dir=/home/mchawla/dbbackup_16_01_2020/inc --incremental-basedir=/home/mchawla/dbbackup_16_01_2020/full -S /home/mchawla/MS_5_7_29/socket.sock --datadir=/home/mchawla/MS_5_7_29/data --lock-ddl-per-table

      Prepare the full backup

      /home/mchawla/pxb_2_4_18_release_debug/bin/xtrabackup --user=root --password='' --prepare --apply-log-only --target_dir=/home/mchawla/dbbackup_16_01_2020/full

      Prepare the incremental backup

      /home/mchawla/pxb_2_4_18_release_debug/bin/xtrabackup --user=root --password='' --prepare --target_dir=/home/mchawla/dbbackup_16_01_2020/full --incremental-dir=/home/mchawla/dbbackup_16_01_2020/inc

      PXB crashes during prepare

      InnoDB: Assertion failure in thread 139990752794368 in file rem0rec.ic line 435
      InnoDB: Failing assertion: ret > 0
      InnoDB: We intentionally generate a memory trap.
      InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
      InnoDB: If you get repeated assertion failures or crashes, even
      InnoDB: immediately after the mysqld startup, there may be
      InnoDB: corruption in the InnoDB tablespace. Please refer to
      InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
      InnoDB: about forcing recovery.
      15:09:50 UTC - xtrabackup got signal 6 ;
      This could be because you hit a bug or data is corrupted.
      This error can also be caused by malfunctioning hardware.
      Attempting to collect some information that could help diagnose the problem.
      As this is a crash and something is definitely wrong, the information
      collection process might fail.
      Thread pointer: 0x0
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0 thread_stack 0x10000
      /home/mchawla/pxb_2_4_18_release_debug/bin/xtrabackup(my_print_stacktrace+0x35)[0x16f5a03]
      /home/mchawla/pxb_2_4_18_release_debug/bin/xtrabackup(handle_fatal_signal+0x2a5)[0x1273bd8]
      /usr/lib64/libpthread.so.0(+0xf5f0)[0x7f522c1755f0]
      /usr/lib64/libc.so.6(gsignal+0x37)[0x7f5229dee337]
      /usr/lib64/libc.so.6(abort+0x148)[0x7f5229defa28]
      /home/mchawla/pxb_2_4_18_release_debug/bin/xtrabackup[0x13414cb]
      /home/mchawla/pxb_2_4_18_release_debug/bin/xtrabackup[0x143cc1b]
      /home/mchawla/pxb_2_4_18_release_debug/bin/xtrabackup(_Z20rec_get_offsets_funcPKhPK12dict_index_tPmmPKcmPP16mem_block_info_t+0x12d)[0x143ee0b]
      /home/mchawla/pxb_2_4_18_release_debug/bin/xtrabackup(_Z25page_cur_parse_insert_recmPKhS0_P11buf_block_tP12dict_index_tP5mtr_t+0x465)[0x147f971]
      /home/mchawla/pxb_2_4_18_release_debug/bin/xtrabackup[0x14a57db]
      /home/mchawla/pxb_2_4_18_release_debug/bin/xtrabackup(_Z22recv_recover_page_funcmP11buf_block_t+0x666)[0x14a75fd]
      /home/mchawla/pxb_2_4_18_release_debug/bin/xtrabackup(_Z20buf_page_io_completeP10buf_page_tb+0x82f)[0x163fe59]
      /home/mchawla/pxb_2_4_18_release_debug/bin/xtrabackup(_Z12fil_aio_waitm+0x1b1)[0x15bb3ab]
      /home/mchawla/pxb_2_4_18_release_debug/bin/xtrabackup(io_handler_thread+0x5b)[0x139399f]
      /usr/lib64/libpthread.so.0(+0x7e65)[0x7f522c16de65]
      /usr/lib64/libc.so.6(clone+0x6d)[0x7f5229eb688d]

      Scenario 2: Full text index is created and dropped
      Create and drop a full text index

      ( for ((i=1; i<=10; i++)); do
       # Check if database is up otherwise exit the loop
       ${mysqldir}/bin//mysqladmin ping --user=root --socket=${mysqldir}/socket.sock 2>/dev/null 1>&2
       if [ "$?" -ne 0 ]; then
       break
       fi
       ${mysqldir}/bin/mysql -uroot -S${mysqldir}/socket.sock -e "CREATE FULLTEXT INDEX full_index on test.sbtest1 (pad);" >/dev/null 2>&1
       ${mysqldir}/bin/mysql -uroot -S${mysqldir}/socket.sock -e "DROP INDEX full_index on test.sbtest1;" >/dev/null 2>&1
       done ) &

      PXB crashes during incremental backup prepare

      2020-01-16 07:21:37 0x7f6582222700 InnoDB: Assertion failure in thread 140073951700736 in file page0page.ic line 754
      InnoDB: Failing assertion: page_offset(rec) <= page_header_get_field(page, PAGE_HEAP_TOP)
      InnoDB: We intentionally generate a memory trap.
      InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
      InnoDB: If you get repeated assertion failures or crashes, even
      InnoDB: immediately after the mysqld startup, there may be
      InnoDB: corruption in the InnoDB tablespace. Please refer to
      InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
      InnoDB: about forcing recovery.
      07:21:37 UTC - xtrabackup got signal 6 ;
      This could be because you hit a bug or data is corrupted.
      This error can also be caused by malfunctioning hardware.
      Attempting to collect some information that could help diagnose the problem.
      As this is a crash and something is definitely wrong, the information
      collection process might fail.
      Thread pointer: 0x0
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0 thread_stack 0x10000
      /home/mchawla/pxb_2_4_18_release_debug/bin/xtrabackup(my_print_stacktrace+0x35)[0x16f5a03]
      /home/mchawla/pxb_2_4_18_release_debug/bin/xtrabackup(handle_fatal_signal+0x2a5)[0x1273bd8]
      /usr/lib64/libpthread.so.0(+0xf5f0)[0x7f658b21e5f0]
      /usr/lib64/libc.so.6(gsignal+0x37)[0x7f6588e97337]
      /usr/lib64/libc.so.6(abort+0x148)[0x7f6588e98a28]
      /home/mchawla/pxb_2_4_18_release_debug/bin/xtrabackup[0x13414cb]
      /home/mchawla/pxb_2_4_18_release_debug/bin/xtrabackup[0x147c23f]
      /home/mchawla/pxb_2_4_18_release_debug/bin/xtrabackup[0x147be74]
      /home/mchawla/pxb_2_4_18_release_debug/bin/xtrabackup(_Z23page_cur_insert_rec_lowPhP12dict_index_tPKhPmP5mtr_t+0x205)[0x147ffa6]
      /home/mchawla/pxb_2_4_18_release_debug/bin/xtrabackup[0x147d39e]
      /home/mchawla/pxb_2_4_18_release_debug/bin/xtrabackup(_Z25page_cur_parse_insert_recmPKhS0_P11buf_block_tP12dict_index_tP5mtr_t+0x802)[0x147fd0e]
      /home/mchawla/pxb_2_4_18_release_debug/bin/xtrabackup[0x14a57db]
      /home/mchawla/pxb_2_4_18_release_debug/bin/xtrabackup(_Z22recv_recover_page_funcmP11buf_block_t+0x666)[0x14a75fd]
      /home/mchawla/pxb_2_4_18_release_debug/bin/xtrabackup(_Z20buf_page_io_completeP10buf_page_tb+0x82f)[0x163fe59]
      /home/mchawla/pxb_2_4_18_release_debug/bin/xtrabackup(_Z12fil_aio_waitm+0x1b1)[0x15bb3ab]
      /home/mchawla/pxb_2_4_18_release_debug/bin/xtrabackup(io_handler_thread+0x5b)[0x139399f]
      /usr/lib64/libpthread.so.0(+0x7e65)[0x7f658b216e65]
      /usr/lib64/libc.so.6(clone+0x6d)[0x7f6588f5f88d]

      Note: This issue is similar to PXB-1824.

      Attachments

        Issue Links

          Activity

            People

              marcelo.altmann Marcelo Altmann
              manish.chawla Manish Chawla
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - Not Specified
                  Not Specified
                  Logged:
                  Time Spent - 1 day, 4 hours
                  1d 4h

                  Smart Checklist