Uploaded image for project: 'Percona XtraBackup'
  1. Percona XtraBackup
  2. PXB-421

LP #1425269: Long-lasting backup of active databases fails with "Combined size of log files must be < 512 GB"

    Details

    • Type: Bug
    • Status: Done
    • Priority: High
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

      **Reported in Launchpad by Eike Frost last update 14-10-2015 21:42:41

      When backing up a somewhat large, disk-backed MariaDB installation with active writes going in (~10TiB in compressed size, backup taking several days to finish, and somewhat busy writes during that time), unforeseen trouble arises due to the limitation of log files to 512GiB; XtraBackup does not warn of this problem when creating the backup (claiming everything went OK), but the backup is unusable due to --apply-log being impossible to do. This is not related to bug #1264432 (while the prepare process does increase this logfile size by 1/8 on every attempt, even the base backup had a logfile of > 512GiB).

      ~~~~
      $ innobackupex --apply-log --export .

      InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy
      and Percona LLC and/or its affiliates 2009-2013. All Rights Reserved.

      This software is published under
      the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.

      Get the latest version of Percona XtraBackup, documentation, and help resources:
      http://www.percona.com/xb/p

      150224 12:10:16 innobackupex: Starting the apply-log operation

      IMPORTANT: Please check that the apply-log run completes successfully.
      At the end of a successful apply-log run innobackupex
      prints "completed OK!".

      150224 12:10:16 innobackupex: Starting ibbackup with command: xtrabackup -defaults-file="/dbdata/tmpbackup/backup-my.cnf" --defaults-group="mysqld" --prepare --target
      dir=/dbdata/tmpbackup --export

      xtrabackup version 2.2.9 based on MySQL server 5.6.22 Linux (x86_64) (revision id: )
      xtrabackup: auto-enabling --innodb-file-per-table due to the --export option
      xtrabackup: cd to /dbdata/tmpbackup
      xtrabackup: This target seems to be not prepared yet.
      xtrabackup: xtrabackup_logfile detected: size=747591827456, start_lsn=(27657099615443)
      xtrabackup: using the following InnoDB configuration for recovery:
      xtrabackup: innodb_data_home_dir = ./
      xtrabackup: innodb_data_file_path = ibdata1:12M:autoextend
      xtrabackup: innodb_log_group_home_dir = ./
      xtrabackup: innodb_log_files_in_group = 1
      xtrabackup: innodb_log_file_size = 747591827456
      xtrabackup: using the following InnoDB configuration for recovery:
      xtrabackup: innodb_data_home_dir = ./
      xtrabackup: innodb_data_file_path = ibdata1:12M:autoextend
      xtrabackup: innodb_log_group_home_dir = ./
      xtrabackup: innodb_log_files_in_group = 1
      xtrabackup: innodb_log_file_size = 747591827456
      xtrabackup: Starting InnoDB instance for recovery.
      xtrabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter)
      InnoDB: Using atomics to ref count buffer pool pages
      InnoDB: The InnoDB memory heap is disabled
      InnoDB: Mutexes and rw_locks use GCC atomic builtins
      InnoDB: Memory barrier is not used
      InnoDB: Compressed tables use zlib 1.2.7
      InnoDB: Using CPU crc32 instructions
      InnoDB: Initializing buffer pool, size = 100.0M
      InnoDB: Completed initialization of buffer pool
      InnoDB: Combined size of log files must be < 512 GB
      xtrabackup: innodb_init(): Error occured.
      innobackupex: got a fatal error with the following stacktrace: at /usr/bin/innobackupex line 2642.
      main::apply_log() called at /usr/bin/innobackupex line 1570
      innobackupex: Error:
      innobackupex: ibbackup failed at /usr/bin/innobackupex line 2642.
      ~~~~

      XtraBackup should, at the very least, inform of this problem when creating the backup and not suggest that everything is OK (only to discover down the line that the backup is unusable for recovery).

      I am not sure whether it is possible to somehow split log-files and apply them one after another to circumvent this limitation in InnoDB.

      Fortunately for me I can partition the backups since this installation uses files_per_table and several databases and thus I can reduce the timeframe where writes need to captured in the log, reducing it to below 455G (accounting for the 1/8 margin required for redo-log-space added at prepare-time). Still this could be a very nasty surprise and is at least annoying when it happens unexpectedly.

        Smart Checklist

          Attachments

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                lpjirasync lpjirasync (Inactive)
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: