Uploaded image for project: 'Percona XtraBackup'
  1. Percona XtraBackup
  2. PXB-503

LP #1690823: streaming backups of large databases fail when receiving side runs pxb 2.3.8

Details

    • Bug
    • Status: Done
    • High
    • Resolution: Fixed
    • None
    • None
    • None

    Description

      **Reported in Launchpad by Rene H. Larsen last update 20-06-2017 08:07:34

      Streaming backups from a database containing many tables via xbstream to a host that has the xbstream from Percona XtraBackup 2.3.8 installed fails. This is due to the receiving xbstream process running out of file descriptors:

      xbstream: Out of resources when opening file './magento/sales_invoice.ibd.qp' (Errcode: 24 - Too many open files)
      xbstream: failed to create file.

      Meanwhile, this is logged to the innobackupex log file:

      innobackupex: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)
      [01] xtrabackup: Error: failed to copy datafile.

      Instrumenting the xbstream binary with strace reveals the cause of the problem:

      [...]
      6578 read(0, "XBSTCK01\0E \0\0\0", 14) = 14
      6578 read(0, "magento/directory_country.ibd.qp", 32) = 32
      6578 read(0, "XBSTCK01\0P\"\0\0\0", 14) = 14
      6578 read(0, "magento/eav_attribute_group.ibd."..., 34) = 34
      6578 read(0, "\n\20\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 16) = 16
      6578 read(0, "\223L\271\25", 4) = 4
      6578 read(0, "qpress10\0\0\1\0\0\0\0\0F\27\0\0\0eav_attribu"..., 4106) = 4106
      6578 mkdir("./magento/", 0700) = -1 EEXIST (File exists)
      6578 open("./magento/eav_attribute_group.ibd.qp", O_WRONLY|O_CREAT|O_EXCL|O_NOFOLLOW, 0660) = 978
      6578 write(978, "qpress10\0\0\1\0\0\0\0\0F\27\0\0\0eav_attribu"..., 4106) = 4106
      6578 fadvise64(978, 0, 0, POSIX_FADV_DONTNEED) = 0
      6578 read(0, "XBSTCK01\0E\"\0\0\0", 14) = 14
      6578 read(0, "magento/eav_attribute_group.ibd."..., 34) = 34
      6578 read(0, "XBSTCK01\0P(\0\0\0", 14) = 14
      6578 read(0, "magento/sales_refunded_aggregate"..., 40) = 40
      6578 read(0, "\307\t\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 16) = 16
      6578 read(0, "i\210\v\313", 4) = 4
      6578 read(0, "qpress10\0\0\1\0\0\0\0\0F\35\0\0\0sales_refun"..., 2503) = 2503
      6578 mkdir("./magento/", 0700) = -1 EEXIST (File exists)
      6578 open("./magento/sales_refunded_aggregated.ibd.qp", O_WRONLY|O_CREAT|O_EXCL|O_NOFOLLOW, 0660) = 979
      6578 write(979, "qpress10\0\0\1\0\0\0\0\0F\35\0\0\0sales_refun"..., 2503) = 2503
      6578 fadvise64(979, 0, 0, POSIX_FADV_DONTNEED) = 0
      [...]

      Note that the created files are never closed. With the xbstream binary from 2.3.7 it looks like this, and it works perfectly:

      [...]
      22980 read(0, "XBSTCK01\0P\"\0\0\0", 14) = 14
      22980 read(0, "magento/eav_attribute_group.ibd."..., 34) = 34
      22980 read(0, "\n\20\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 16) = 16
      22980 read(0, "\223L\271\25", 4) = 4
      22980 read(0, "qpress10\0\0\1\0\0\0\0\0F\27\0\0\0eav_attribu"..., 4106) = 4106
      22980 mkdir("./magento/", 0700) = -1 EEXIST (File exists)
      22980 open("./magento/eav_attribute_group.ibd.qp", O_WRONLY|O_CREAT|O_EXCL|O_NOFOLLOW, 0660) = 3
      22980 write(3, "qpress10\0\0\1\0\0\0\0\0F\27\0\0\0eav_attribu"..., 4106) = 4106
      22980 fadvise64(3, 0, 0, POSIX_FADV_DONTNEED) = 0
      22980 read(0, "XBSTCK01\0E\"\0\0\0", 14) = 14
      22980 read(0, "magento/eav_attribute_group.ibd."..., 34) = 34
      22980 fsync(3) = 0
      22980 close(3) = 0
      22980 read(0, "XBSTCK01\0P(\0\0\0", 14) = 14
      22980 read(0, "magento/sales_refunded_aggregate"..., 40) = 40
      22980 read(0, "\307\t\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 16) = 16
      22980 read(0, "i\210\v\313", 4) = 4
      22980 read(0, "qpress10\0\0\1\0\0\0\0\0F\35\0\0\0sales_refun"..., 2503) = 2503
      22980 mkdir("./magento/", 0700) = -1 EEXIST (File exists)
      22980 open("./magento/sales_refunded_aggregated.ibd.qp", O_WRONLY|O_CREAT|O_EXCL|O_NOFOLLOW, 0660) = 3
      22980 write(3, "qpress10\0\0\1\0\0\0\0\0F\35\0\0\0sales_refun"..., 2503) = 2503
      22980 fadvise64(3, 0, 0, POSIX_FADV_DONTNEED) = 0
      22980 read(0, "XBSTCK01\0E(\0\0\0", 14) = 14
      22980 read(0, "magento/sales_refunded_aggregate"..., 40) = 40
      22980 fsync(3) = 0
      22980 close(3) = 0
      [...]

      The exact package versions I'm testing with (on Debian wheezy):
      2.3.7-2.wheezy
      2.3.8-1.wheezy

      For now, I've reverted to version 2.3.7 on the receiving side.

      Attachments

        Activity

          People

            Unassigned Unassigned
            lpjirasync lpjirasync (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Smart Checklist