Uploaded image for project: 'Percona XtraBackup'
  1. Percona XtraBackup
  2. PXB-1552

Preparing an incremental backup will crash if compressed InnoDB undo tablespaces are not removed beforehand

    Details

    • Type: Bug
    • Status: Done
    • Priority: Medium
    • Resolution: Fixed
    • Affects Version/s: 2.4.11, Not 8.0
    • Fix Version/s: 2.4.14
    • Component/s: innobackupex script
    • Labels:
      None

      Description

      To reproduce, MySQL should start with this enabled:

       

      --innodb_undo_tablespaces=4 
      

      Create a backup and incremental:

       

      innobackupex --defaults-file=/home/user/sandboxes/msb_5_7_22/my.sandbox.cnf --host=127.0.0.1 --port=5722 --user=root --password=msandbox --no-timestamp /home/user/backups/full --compress ;
      innobackupex --defaults-file=/home/user/sandboxes/msb_5_7_22/my.sandbox.cnf --host=127.0.0.1 --port=5722 --user=root --password=msandbox --no-timestamp --incremental /home/user/backups/inc1 --incremental-basedir=/home/user/backups/full --compress ;

      Decompress:

      innobackupex --decompress /home/user/backups/full && innobackupex --decompress /home/user/backups/inc1 ;

      Prepare:

      innobackupex --apply-log --redo-only /home/user/backups/full ;
      innobackupex --apply-log --redo-only /home/user/backups/full --incremental-dir=/home/user/backups/inc1 ;

      The 2nd prepare will fail with this error:

       

      encryption: using gcrypt 1.8.1
      180621 05:42:19 innobackupex: Starting the apply-log operation
      IMPORTANT: Please check that the apply-log run completes successfully.
                 At the end of a successful apply-log run innobackupex
                 prints "completed OK!".
      innobackupex version 2.4.11 based on MySQL server 5.7.19 Linux (x86_64) (revision id: b4e0db5)
      incremental backup from 2585807 is enabled.
      xtrabackup: cd to /home/user/backups/full/
      xtrabackup: This target seems to be already prepared with --apply-log-only.
      InnoDB: Number of pools: 1
      xtrabackup: xtrabackup_logfile detected: size=252116992, start_lsn=(2585807)
      xtrabackup: using the following InnoDB configuration for recovery:
      xtrabackup:   innodb_data_home_dir = .
      xtrabackup:   innodb_data_file_path = ibdata1:12M:autoextend
      xtrabackup:   innodb_log_group_home_dir = /home/user/backups/inc1/
      xtrabackup:   innodb_log_files_in_group = 1
      xtrabackup:   innodb_log_file_size = 252116992
      2018-06-21 05:42:19 0x7f9c9f90c740  InnoDB: Assertion failure in thread 140310668691264 in file srv0start.cc line 961
      InnoDB: Failing assertion: prev_space_id + 1 == undo_tablespace_ids[i]
      InnoDB: We intentionally generate a memory trap.
      InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
      InnoDB: If you get repeated assertion failures or crashes, even
      InnoDB: immediately after the mysqld startup, there may be
      InnoDB: corruption in the InnoDB tablespace. Please refer to
      InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
      InnoDB: about forcing recovery.
      05:42:19 UTC - xtrabackup got signal 6 ;
      This could be because you hit a bug or data is corrupted.
      This error can also be caused by malfunctioning hardware.
      Attempting to collect some information that could help diagnose the problem.
      As this is a crash and something is definitely wrong, the information
      collection process might fail.
      Thread pointer: 0x0
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0 thread_stack 0x10000
      innobackupex(my_print_stacktrace+0x3b)[0x5569578a125b]
      innobackupex(handle_fatal_signal+0x293)[0x556957710b83]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7f9c9f4ea890]
      /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7f9c9d381e97]
      /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7f9c9d383801]
      innobackupex(+0x567b3d)[0x556956fabb3d]
      innobackupex(_Z25srv_undo_tablespaces_initbbmPm+0x840)[0x556957239dd0]
      innobackupex(_Z18xb_data_files_initv+0x16d)[0x556956fcb1dd]
      innobackupex(+0x58c10d)[0x556956fd010d]
      innobackupex(main+0x818)[0x556956fb0258]
      /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7)[0x7f9c9d364b97]
      innobackupex(_start+0x2a)[0x556956fc418a]
      

       

      The culprit

       

      ls full/undo00*.qp
      full/undo001.qp  full/undo002.qp  full/undo003.qp  full/undo004.qp
      

       

      The workaround is to delete all .qp files prior to preparing the backup

      for bf in `find /home/user/backups/full -iname "*\.qp"`; do ls -lh $bf;rm -fv $bf; done
      for bf in `find /home/user/backups/inc1 -iname "*\.qp"`; do ls -lh $bf;rm -fv $bf; done
      innobackupex --apply-log --redo-only /home/user/backups/full ;
      innobackupex --apply-log --redo-only /home/user/backups/full --incremental-dir=/home/user/backups/inc1 ;

        Smart Checklist

          Attachments

            Activity

              People

              • Assignee:
                sergei.glushchenko Sergei Glushchenko
                Reporter:
                jaime.sicam@percona.com Jaime Sicam
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - Not Specified
                  Not Specified
                  Logged:
                  Time Spent - 4 hours, 8 minutes
                  4h 8m