LP #1690823: streaming backups of large databases fail when receiving side runs pxb 2.3.8

**Reported in Launchpad by Rene H. Larsen last update 20-06-2017 08:07:34

Streaming backups from a database containing many tables via xbstream to a host that has the xbstream from Percona XtraBackup 2.3.8 installed fails. This is due to the receiving xbstream process running out of file descriptors:

xbstream: Out of resources when opening file './magento/sales_invoice.ibd.qp' (Errcode: 24 - Too many open files)
xbstream: failed to create file.

Meanwhile, this is logged to the innobackupex log file:

innobackupex: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)
[01] xtrabackup: Error: failed to copy datafile.

Instrumenting the xbstream binary with strace reveals the cause of the problem:

[...]
6578 read(0, "XBSTCK01\0E \0\0\0", 14) = 14
6578 read(0, "magento/directory_country.ibd.qp", 32) = 32
6578 read(0, "XBSTCK01\0P\"\0\0\0", 14) = 14
6578 read(0, "magento/eav_attribute_group.ibd."..., 34) = 34
6578 read(0, "\n\20\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 16) = 16
6578 read(0, "\223L\271\25", 4) = 4
6578 read(0, "qpress10\0\0\1\0\0\0\0\0F\27\0\0\0eav_attribu"..., 4106) = 4106
6578 mkdir("./magento/", 0700) = -1 EEXIST (File exists)
6578 open("./magento/eav_attribute_group.ibd.qp", O_WRONLY|O_CREAT|O_EXCL|O_NOFOLLOW, 0660) = 978
6578 write(978, "qpress10\0\0\1\0\0\0\0\0F\27\0\0\0eav_attribu"..., 4106) = 4106
6578 fadvise64(978, 0, 0, POSIX_FADV_DONTNEED) = 0
6578 read(0, "XBSTCK01\0E\"\0\0\0", 14) = 14
6578 read(0, "magento/eav_attribute_group.ibd."..., 34) = 34
6578 read(0, "XBSTCK01\0P(\0\0\0", 14) = 14
6578 read(0, "magento/sales_refunded_aggregate"..., 40) = 40
6578 read(0, "\307\t\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 16) = 16
6578 read(0, "i\210\v\313", 4) = 4
6578 read(0, "qpress10\0\0\1\0\0\0\0\0F\35\0\0\0sales_refun"..., 2503) = 2503
6578 mkdir("./magento/", 0700) = -1 EEXIST (File exists)
6578 open("./magento/sales_refunded_aggregated.ibd.qp", O_WRONLY|O_CREAT|O_EXCL|O_NOFOLLOW, 0660) = 979
6578 write(979, "qpress10\0\0\1\0\0\0\0\0F\35\0\0\0sales_refun"..., 2503) = 2503
6578 fadvise64(979, 0, 0, POSIX_FADV_DONTNEED) = 0
[...]

Note that the created files are never closed. With the xbstream binary from 2.3.7 it looks like this, and it works perfectly:

[...]
22980 read(0, "XBSTCK01\0P\"\0\0\0", 14) = 14
22980 read(0, "magento/eav_attribute_group.ibd."..., 34) = 34
22980 read(0, "\n\20\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 16) = 16
22980 read(0, "\223L\271\25", 4) = 4
22980 read(0, "qpress10\0\0\1\0\0\0\0\0F\27\0\0\0eav_attribu"..., 4106) = 4106
22980 mkdir("./magento/", 0700) = -1 EEXIST (File exists)
22980 open("./magento/eav_attribute_group.ibd.qp", O_WRONLY|O_CREAT|O_EXCL|O_NOFOLLOW, 0660) = 3
22980 write(3, "qpress10\0\0\1\0\0\0\0\0F\27\0\0\0eav_attribu"..., 4106) = 4106
22980 fadvise64(3, 0, 0, POSIX_FADV_DONTNEED) = 0
22980 read(0, "XBSTCK01\0E\"\0\0\0", 14) = 14
22980 read(0, "magento/eav_attribute_group.ibd."..., 34) = 34
22980 fsync(3) = 0
22980 close(3) = 0
22980 read(0, "XBSTCK01\0P(\0\0\0", 14) = 14
22980 read(0, "magento/sales_refunded_aggregate"..., 40) = 40
22980 read(0, "\307\t\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 16) = 16
22980 read(0, "i\210\v\313", 4) = 4
22980 read(0, "qpress10\0\0\1\0\0\0\0\0F\35\0\0\0sales_refun"..., 2503) = 2503
22980 mkdir("./magento/", 0700) = -1 EEXIST (File exists)
22980 open("./magento/sales_refunded_aggregated.ibd.qp", O_WRONLY|O_CREAT|O_EXCL|O_NOFOLLOW, 0660) = 3
22980 write(3, "qpress10\0\0\1\0\0\0\0\0F\35\0\0\0sales_refun"..., 2503) = 2503
22980 fadvise64(3, 0, 0, POSIX_FADV_DONTNEED) = 0
22980 read(0, "XBSTCK01\0E(\0\0\0", 14) = 14
22980 read(0, "magento/sales_refunded_aggregate"..., 40) = 40
22980 fsync(3) = 0
22980 close(3) = 0
[...]

The exact package versions I'm testing with (on Debian wheezy):
2.3.7-2.wheezy
2.3.8-1.wheezy

For now, I've reverted to version 2.3.7 on the receiving side.