Details
-
Bug
-
Status: Done
-
Medium
-
Resolution: Duplicate
-
2.4.12
-
None
-
None
-
None
Description
Hello.
I am trying to add a new node to a Percona Cluster but the SST fails after a few seconds while processing one of the tables of our database.
Where are using Ubuntu 16.04 on the donor, with this Percona cluster version:
5.7.23-23-57-log - Percona XtraDB Cluster (GPL), Release rel23, Revision f5578f0, WSREP version 31.31, wsrep_31.31
And in the joiner we are using 18.04 with the Percona version that ships with Ubuntu 18.04: 5.7.20-29.24-0ubuntu2.1
This is the dump that we see on innobackup.backup.log:
...
181011 13:38:26 >> log scanned up to (30938294862763)
181011 13:38:27 >> log scanned up to (30938294862763)
181011 13:38:28 >> log scanned up to (30938295065837)
181011 13:38:29 >> log scanned up to (30938295102480)
xtrabackup: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)
xb_stream_write_data() failed.
xtrabackup: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)
xb_stream_write_data() failed.
xtrabackup: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)
xb_stream_write_data() failed.
xtrabackup: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)
[03] xtrabackup: Error: xtrabackup_copy_datafile() failed.
[03] xtrabackup: Error: failed to copy datafile.
xtrabackup: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)
[04] xtrabackup: Error: xtrabackup_copy_datafile() failed.
[04] xtrabackup: Error: failed to copy datafile.
xtrabackup: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)
-
-
- Error in `xtrabackup': corrupted size vs. prev_size: 0x0000000002976360 ***
[01] xtrabackup: Error: xtrabackup_copy_datafile() failed.
[01] xtrabackup: Error: failed to copy datafile.
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7f292efdf7e5]
/lib/x86_64-linux-gnu/libc.so.6(+0x80dfb)[0x7f292efe8dfb]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7f292efec53c]
/lib/x86_64-linux-gnu/libc.so.6(+0x3a025)[0x7f292efa2025]
/lib/x86_64-linux-gnu/libc.so.6(+0x3a045)[0x7f292efa2045]
xtrabackup[0x6fd4c7]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f2930fc86ba]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f292f06f41d]
======= Memory map: ========
00400000-016e2000 r-xp 00000000 09:02 27132532 /usr/bin/xtrabackup
018e1000-019a9000 r--p 012e1000 09:02 27132532 /usr/bin/xtrabackup
019a9000-01a55000 rw-p 013a9000 09:02 27132532 /usr/bin/xtrabackup
01a55000-01ad9000 rw-p 00000000 00:00 0
0295e000-02d2c000 rw-p 00000000 00:00 0 [heap]
7f28f4000000-7f28f4021000 rw-p 00000000 00:00 0
7f28f4021000-7f28f8000000 ---p 00000000 00:00 0
7f28fc000000-7f28fd40d000 rw-p 00000000 00:00 0
7f28fd40d000-7f2900000000 ---p 00000000 00:00 0
7f2904000000-7f290540d000 rw-p 00000000 00:00 0
7f290540d000-7f2908000000 ---p 00000000 00:00 0
7f290c000000-7f290c021000 rw-p 00000000 00:00 0
7f290c021000-7f2910000000 ---p 00000000 00:00 0
7f2910000000-7f2910041000 rw-p 00000000 00:00 0
7f2910041000-7f2914000000 ---p 00000000 00:00 0
7f2914000000-7f2914021000 rw-p 00000000 00:00 0
7f2914021000-7f2918000000 ---p 00000000 00:00 0
7f2918000000-7f291940d000 rw-p 00000000 00:00 0
7f291940d000-7f291c000000 ---p 00000000 00:00 0
7f291c000000-7f291c021000 rw-p 00000000 00:00 0
7f291c021000-7f2920000000 ---p 00000000 00:00 0
7f29212ab000-7f29212ac000 ---p 00000000 00:00 0
7f29212ac000-7f2921aac000 rw-p 00000000 00:00 0
7f29237ff000-7f2923800000 ---p 00000000 00:00 0
7f2923800000-7f2924000000 rw-p 00000000 00:00 0
7f2924000000-7f2924021000 rw-p 00000000 00:00 0
7f2924021000-7f2928000000 ---p 00000000 00:00 0
7f29280a4000-7f29280a5000 ---p 00000000 00:00 0
7f29280a5000-7f29288a5000 rw-p 00000000 00:00 0
7f29288a5000-7f29288a6000 ---p 00000000 00:00 0
7f29288a6000-7f29290a6000 rw-p 00000000 00:00 0
7f29290a6000-7f29290a7000 ---p 00000000 00:00 0
7f29290a7000-7f2929aac000 rw-p 00000000 00:00 0
7f2929aac000-7f2929aad000 ---p 00000000 00:00 0
7f2929aad000-7f292a2ad000 rw-p 00000000 00:00 0
7f292a2ad000-7f292a2ae000 ---p 00000000 00:00 0
7f292a2ae000-7f292aaae000 rw-p 00000000 00:00 0
7f292aaae000-7f292aaaf000 ---p 00000000 00:00 0
7f292aaaf000-7f292b2af000 rw-p 00000000 00:00 0
7f292b2af000-7f292b2b0000 ---p 00000000 00:00 0
7f292b2b0000-7f292eb42000 rw-p 00000000 00:00 0
7f292eb42000-7f292eb4d000 r-xp 00000000 09:02 19923164 /lib/x86_64-linux-gnu/libnss_files-2.23.so
7f292eb4d000-7f292ed4c000 ---p 0000b000 09:02 19923164 /lib/x86_64-linux-gnu/libnss_files-2.23.so
7f292ed4c000-7f292ed4d000 r--p 0000a000 09:02 19923164 /lib/x86_64-linux-gnu/libnss_files-2.23.so
7f292ed4d000-7f292ed4e000 rw-p 0000b000 09:02 19923164 /lib/x86_64-linux-gnu/libnss_files-2.23.so
7f292ed4e000-7f292ed54000 rw-p 00000000 00:00 0
7f292ed54000-7f292ed66000 r-xp 00000000 09:02 19926551 /lib/x86_64-linux-gnu/libgpg-error.so.0.17.0
7f292ed66000-7f292ef66000 ---p 00012000 09:02 19926551 /lib/x86_64-linux-gnu/libgpg-error.so.0.17.0
7f292ef66000-7f292ef67000 r--p 00012000 09:02 19926551 /lib/x86_64-linux-gnu/libgpg-error.so.0.17.0
7f292ef67000-7f292ef68000 rw-p 00013000 09:02 19926551 /lib/x86_64-linux-gnu/libgpg-error.so.0.17.0
7f292ef68000-7f292f128000 r-xp 00000000 09:02 19923151 /lib/x86_64-linux-gnu/libc-2.23.so
7f292f128000-7f292f328000 ---p 001c0000 09:02 19923151 /lib/x86_64-linux-gnu/libc-2.23.so
7f292f328000-7f292f32c000 r--p 001c0000 09:02 19923151 /lib/x86_64-linux-gnu/libc-2.23.so
7f292f32c000-7f292f32e000 rw-p 001c4000 09:02 19923151 /lib/x86_64-linux-gnu/libc-2.23.so
7f292f32e000-7f292f332000 rw-p 00000000 00:00 0
7f292f332000-7f292f348000 r-xp 00000000 09:02 19926535 /lib/x86_64-linux-gnu/libgcc_s.so.1
7f292f348000-7f292f547000 ---p 00016000 09:02 19926535 /lib/x86_64-linux-gnu/libgcc_s.so.1
7f292f547000-7f292f548000 rw-p 00015000 09:02 19926535 /lib/x86_64-linux-gnu/libgcc_s.so.1
7f292f548000-7f292f650000 r-xp 00000000 09:02 19923009 /lib/x86_64-linux-gnu/libm-2.23.so
7f292f650000-7f292f84f000 ---p 00108000 09:02 19923009 /lib/x86_64-linux-gnu/libm-2.23.so
7f292f84f000-7f292f850000 r--p 00107000 09:02 19923009 /lib/x86_64-linux-gnu/libm-2.23.so
7f292f850000-7f292f851000 rw-p 00108000 09:02 19923009 /lib/x86_64-linux-gnu/libm-2.23.so
7f292f851000-7f292f9c3000 r-xp 00000000 09:02 27395008 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7f292f9c3000-7f292fbc3000 ---p 00172000 09:02 27395008 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7f292fbc3000-7f292fbcd000 r--p 00172000 09:02 27395008 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7f292fbcd000-7f292fbcf000 rw-p 0017c000 09:02 27395008 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
7f292fbcf000-7f292fbd3000 rw-p 00000000 00:00 0
7f292fbd3000-7f292fbd4000 r-xp 00000000 09:02 19926412 /lib/x86_64-linux-gnu/libaio.so.1.0.1
7f292fbd4000-7f292fdd3000 ---p 00001000 09:02 19926412 /lib/x86_64-linux-gnu/libaio.so.1.0.1
7f292fdd3000-7f292fdd4000 r--p 00000000 09:02 19926412 /lib/x86_64-linux-gnu/libaio.so.1.0.1
7f292fdd4000-7f292fdd5000 rw-p 00001000 09:02 19926412 /lib/x86_64-linux-gnu/libaio.so.1.0.1
7f292fdd5000-7f292fdee000 r-xp 00000000 09:02 19922961 /lib/x86_64-linux-gnu/libz.so.1.2.8
7f292fdee000-7f292ffed000 ---p 00019000 09:02 19922961 /lib/x86_64-linux-gnu/libz.so.1.2.8
7f292ffed000-7f292ffee000 r--p 00018000 09:02 19922961 /lib/x86_64-linux-gnu/libz.so.1.2.8
7f292ffee000-7f292ffef000 rw-p 00019000 09:02 19922961 /lib/x86_64-linux-gnu/libz.so.1.2.8
7f292ffef000-7f292fff8000 r-xp 00000000 09:02 19923170 /lib/x86_64-linux-gnu/libcrypt-2.23.so
7f292fff8000-7f29301f7000 ---p 00009000 09:02 19923170 /lib/x86_64-linux-gnu/libcrypt-2.23.so
7f29301f7000-7f29301f8000 r--p 00008000 09:02 19923170 /lib/x86_64-linux-gnu/libcrypt-2.23.so
7f29301f8000-7f29301f9000 rw-p 00009000 09:02 19923170 /lib/x86_64-linux-gnu/libcrypt-2.23.so
7f29301f9000-7f2930227000 rw-p 00000000 00:00 0
7f2930227000-7f293022a000 r-xp 00000000 09:02 19923153 /lib/x86_64-linux-gnu/libdl-2.23.so
7f293022a000-7f2930429000 ---p 00003000 09:02 19923153 /lib/x86_64-linux-gnu/libdl-2.23.so
7f2930429000-7f293042a000 r--p 00002000 09:02 19923153 /lib/x86_64-linux-gnu/libdl-2.23.so
7f293042a000-7f293042b000 rw-p 00003000 09:02 19923153 /lib/x86_64-linux-gnu/libdl-2.23.so
7f293042b000-7f2930645000 r-xp 00000000 09:02 19922986 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7f2930645000-7f2930844000 ---p 0021a000 09:02 19922986 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7f2930844000-7f2930860000 r--p 00219000 09:02 19922986 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7f2930860000-7f293086c000 rw-p 00235000 09:02 19922986 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
7f293086c000-7f293086f000 rw-p 00000000 00:00 0
7f293086f000-7f29308cd000 r-xp 00000000 09:02 19922987 /lib/x86_64-linux-gnu/libssl.so.1.0.0
7f29308cd000-7f2930acd000 ---p 0005e000 09:02 19922987 /lib/x86_64-linux-gnu/libssl.so.1.0.0
7f2930acd000-7f2930ad1000 r--p 0005e000 09:02 19922987 /lib/x86_64-linux-gnu/libssl.so.1.0.0
7f2930ad1000-7f2930ad8000 rw-p 00062000 09:02 19922987 /lib/x86_64-linux-gnu/libssl.so.1.0.0
7f2930ad8000-7f2930adf000 r-xp 00000000 09:02 19923169 /lib/x86_64-linux-gnu/librt-2.23.so
7f2930adf000-7f2930cde000 ---p 00007000 09:02 19923169 /lib/x86_64-linux-gnu/librt-2.23.so
7f2930cde000-7f2930cdf000 r--p 00006000 09:02 19923169 /lib/x86_64-linux-gnu/librt-2.23.so
7f2930cdf000-7f2930ce0000 rw-p 00007000 09:02 19923169 /lib/x86_64-linux-gnu/librt-2.23.so
7f2930ce0000-7f2930db7000 r-xp 00000000 09:02 19922984 /lib/x86_64-linux-gnu/libgcrypt.so.20.0.5
7f2930db7000-7f2930fb7000 ---p 000d7000 09:02 19922984 /lib/x86_64-linux-gnu/libgcrypt.so.20.0.5
7f2930fb7000-7f2930fb8000 r--p 000d7000 09:02 19922984 /lib/x86_64-linux-gnu/libgcrypt.so.20.0.5
7f2930fb8000-7f2930fc0000 rw-p 000d8000 09:02 19922984 /lib/x86_64-linux-gnu/libgcrypt.so.20.0.5
7f2930fc0000-7f2930fc1000 rw-p 00000000 00:00 0
7f2930fc1000-7f2930fd9000 r-xp 00000000 09:02 19923150 /lib/x86_64-linux-gnu/libpthread-2.23.so
7f2930fd9000-7f29311d8000 ---p 00018000 09:02 19923150 /lib/x86_64-linux-gnu/libpthread-2.23.so
7f29311d8000-7f29311d9000 r--p 00017000 09:02 19923150 /lib/x86_64-linux-gnu/libpthread-2.23.so
7f29311d9000-7f29311da000 rw-p 00018000 09:02 19923150 /lib/x86_64-linux-gnu/libpthread-2.23.so
7f29311da000-7f29311de000 rw-p 00000000 00:00 0
7f29311de000-7f2931204000 r-xp 00000000 09:02 19923075 /lib/x86_64-linux-gnu/ld-2.23.so
7f2931208000-7f29313fd000 rw-p 00000000 00:00 0
7f2931402000-7f2931403000 rw-p 00000000 00:00 0
7f2931403000-7f2931404000 r--p 00025000 09:02 19923075 /lib/x86_64-linux-gnu/ld-2.23.so
7f2931404000-7f2931405000 rw-p 00026000 09:02 19923075 /lib/x86_64-linux-gnu/ld-2.23.so
7f2931405000-7f2931406000 rw-p 00000000 00:00 0
7ffc5fa39000-7ffc5fa5a000 rw-p 00000000 00:00 0 [stack]
7ffc5fab0000-7ffc5fab3000 r--p 00000000 00:00 0 [vvar]
7ffc5fab3000-7ffc5fab5000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
11:38:30 UTC - xtrabackup got signal 6 ;
This could be because you hit a bug or data is corrupted.
This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.
- Error in `xtrabackup': corrupted size vs. prev_size: 0x0000000002976360 ***
-
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x10000
xtrabackup(my_print_stacktrace+0x3b)[0xc40b9b]
xtrabackup(handle_fatal_signal+0x281)[0xae2761]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f2930fd2390]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x38)[0x7f292ef9d428]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x16a)[0x7f292ef9f02a]
/lib/x86_64-linux-gnu/libc.so.6(+0x777ea)[0x7f292efdf7ea]
/lib/x86_64-linux-gnu/libc.so.6(+0x80dfb)[0x7f292efe8dfb]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7f292efec53c]
/lib/x86_64-linux-gnu/libc.so.6(+0x3a025)[0x7f292efa2025]
/lib/x86_64-linux-gnu/libc.so.6(+0x3a045)[0x7f292efa2045]
xtrabackup[0x6fd4c7]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f2930fc86ba]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f292f06f41d]
Please report a bug at https://jira.percona.com/projects/PXB