Uploaded image for project: 'Percona XtraBackup'
  1. Percona XtraBackup
  2. PXB-1677

Cannot add a new node to a cluster because SST fails

Details

    • Bug
    • Status: Done
    • Medium
    • Resolution: Duplicate
    • 2.4.12
    • None
    • None
    • None

    Description

      Hello.

      I am trying to add a new node to a Percona Cluster but the SST fails after a few seconds while processing one of the tables of our database.

       

      Where are using Ubuntu 16.04 on the donor, with this Percona cluster version:

      5.7.23-23-57-log - Percona XtraDB Cluster (GPL), Release rel23, Revision f5578f0, WSREP version 31.31, wsrep_31.31

       

      And in the joiner we are using 18.04 with the Percona version that ships with Ubuntu 18.04:   5.7.20-29.24-0ubuntu2.1

       

      This is the dump that we see on innobackup.backup.log:

      ...
      181011 13:38:26 >> log scanned up to (30938294862763)
      181011 13:38:27 >> log scanned up to (30938294862763)
      181011 13:38:28 >> log scanned up to (30938295065837)
      181011 13:38:29 >> log scanned up to (30938295102480)
      xtrabackup: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)
      xb_stream_write_data() failed.
      xtrabackup: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)
      xb_stream_write_data() failed.
      xtrabackup: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)
      xb_stream_write_data() failed.
      xtrabackup: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)
      [03] xtrabackup: Error: xtrabackup_copy_datafile() failed.
      [03] xtrabackup: Error: failed to copy datafile.
      xtrabackup: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)
      [04] xtrabackup: Error: xtrabackup_copy_datafile() failed.
      [04] xtrabackup: Error: failed to copy datafile.
      xtrabackup: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)

          • Error in `xtrabackup': corrupted size vs. prev_size: 0x0000000002976360 ***
            [01] xtrabackup: Error: xtrabackup_copy_datafile() failed.
            [01] xtrabackup: Error: failed to copy datafile.
            ======= Backtrace: =========
            /lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7f292efdf7e5]
            /lib/x86_64-linux-gnu/libc.so.6(+0x80dfb)[0x7f292efe8dfb]
            /lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7f292efec53c]
            /lib/x86_64-linux-gnu/libc.so.6(+0x3a025)[0x7f292efa2025]
            /lib/x86_64-linux-gnu/libc.so.6(+0x3a045)[0x7f292efa2045]
            xtrabackup[0x6fd4c7]
            /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f2930fc86ba]
            /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f292f06f41d]
            ======= Memory map: ========
            00400000-016e2000 r-xp 00000000 09:02 27132532 /usr/bin/xtrabackup
            018e1000-019a9000 r--p 012e1000 09:02 27132532 /usr/bin/xtrabackup
            019a9000-01a55000 rw-p 013a9000 09:02 27132532 /usr/bin/xtrabackup
            01a55000-01ad9000 rw-p 00000000 00:00 0
            0295e000-02d2c000 rw-p 00000000 00:00 0 [heap]
            7f28f4000000-7f28f4021000 rw-p 00000000 00:00 0
            7f28f4021000-7f28f8000000 ---p 00000000 00:00 0
            7f28fc000000-7f28fd40d000 rw-p 00000000 00:00 0
            7f28fd40d000-7f2900000000 ---p 00000000 00:00 0
            7f2904000000-7f290540d000 rw-p 00000000 00:00 0
            7f290540d000-7f2908000000 ---p 00000000 00:00 0
            7f290c000000-7f290c021000 rw-p 00000000 00:00 0
            7f290c021000-7f2910000000 ---p 00000000 00:00 0
            7f2910000000-7f2910041000 rw-p 00000000 00:00 0
            7f2910041000-7f2914000000 ---p 00000000 00:00 0
            7f2914000000-7f2914021000 rw-p 00000000 00:00 0
            7f2914021000-7f2918000000 ---p 00000000 00:00 0
            7f2918000000-7f291940d000 rw-p 00000000 00:00 0
            7f291940d000-7f291c000000 ---p 00000000 00:00 0
            7f291c000000-7f291c021000 rw-p 00000000 00:00 0
            7f291c021000-7f2920000000 ---p 00000000 00:00 0
            7f29212ab000-7f29212ac000 ---p 00000000 00:00 0
            7f29212ac000-7f2921aac000 rw-p 00000000 00:00 0
            7f29237ff000-7f2923800000 ---p 00000000 00:00 0
            7f2923800000-7f2924000000 rw-p 00000000 00:00 0
            7f2924000000-7f2924021000 rw-p 00000000 00:00 0
            7f2924021000-7f2928000000 ---p 00000000 00:00 0
            7f29280a4000-7f29280a5000 ---p 00000000 00:00 0
            7f29280a5000-7f29288a5000 rw-p 00000000 00:00 0
            7f29288a5000-7f29288a6000 ---p 00000000 00:00 0
            7f29288a6000-7f29290a6000 rw-p 00000000 00:00 0
            7f29290a6000-7f29290a7000 ---p 00000000 00:00 0
            7f29290a7000-7f2929aac000 rw-p 00000000 00:00 0
            7f2929aac000-7f2929aad000 ---p 00000000 00:00 0
            7f2929aad000-7f292a2ad000 rw-p 00000000 00:00 0
            7f292a2ad000-7f292a2ae000 ---p 00000000 00:00 0
            7f292a2ae000-7f292aaae000 rw-p 00000000 00:00 0
            7f292aaae000-7f292aaaf000 ---p 00000000 00:00 0
            7f292aaaf000-7f292b2af000 rw-p 00000000 00:00 0
            7f292b2af000-7f292b2b0000 ---p 00000000 00:00 0
            7f292b2b0000-7f292eb42000 rw-p 00000000 00:00 0
            7f292eb42000-7f292eb4d000 r-xp 00000000 09:02 19923164 /lib/x86_64-linux-gnu/libnss_files-2.23.so
            7f292eb4d000-7f292ed4c000 ---p 0000b000 09:02 19923164 /lib/x86_64-linux-gnu/libnss_files-2.23.so
            7f292ed4c000-7f292ed4d000 r--p 0000a000 09:02 19923164 /lib/x86_64-linux-gnu/libnss_files-2.23.so
            7f292ed4d000-7f292ed4e000 rw-p 0000b000 09:02 19923164 /lib/x86_64-linux-gnu/libnss_files-2.23.so
            7f292ed4e000-7f292ed54000 rw-p 00000000 00:00 0
            7f292ed54000-7f292ed66000 r-xp 00000000 09:02 19926551 /lib/x86_64-linux-gnu/libgpg-error.so.0.17.0
            7f292ed66000-7f292ef66000 ---p 00012000 09:02 19926551 /lib/x86_64-linux-gnu/libgpg-error.so.0.17.0
            7f292ef66000-7f292ef67000 r--p 00012000 09:02 19926551 /lib/x86_64-linux-gnu/libgpg-error.so.0.17.0
            7f292ef67000-7f292ef68000 rw-p 00013000 09:02 19926551 /lib/x86_64-linux-gnu/libgpg-error.so.0.17.0
            7f292ef68000-7f292f128000 r-xp 00000000 09:02 19923151 /lib/x86_64-linux-gnu/libc-2.23.so
            7f292f128000-7f292f328000 ---p 001c0000 09:02 19923151 /lib/x86_64-linux-gnu/libc-2.23.so
            7f292f328000-7f292f32c000 r--p 001c0000 09:02 19923151 /lib/x86_64-linux-gnu/libc-2.23.so
            7f292f32c000-7f292f32e000 rw-p 001c4000 09:02 19923151 /lib/x86_64-linux-gnu/libc-2.23.so
            7f292f32e000-7f292f332000 rw-p 00000000 00:00 0
            7f292f332000-7f292f348000 r-xp 00000000 09:02 19926535 /lib/x86_64-linux-gnu/libgcc_s.so.1
            7f292f348000-7f292f547000 ---p 00016000 09:02 19926535 /lib/x86_64-linux-gnu/libgcc_s.so.1
            7f292f547000-7f292f548000 rw-p 00015000 09:02 19926535 /lib/x86_64-linux-gnu/libgcc_s.so.1
            7f292f548000-7f292f650000 r-xp 00000000 09:02 19923009 /lib/x86_64-linux-gnu/libm-2.23.so
            7f292f650000-7f292f84f000 ---p 00108000 09:02 19923009 /lib/x86_64-linux-gnu/libm-2.23.so
            7f292f84f000-7f292f850000 r--p 00107000 09:02 19923009 /lib/x86_64-linux-gnu/libm-2.23.so
            7f292f850000-7f292f851000 rw-p 00108000 09:02 19923009 /lib/x86_64-linux-gnu/libm-2.23.so
            7f292f851000-7f292f9c3000 r-xp 00000000 09:02 27395008 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
            7f292f9c3000-7f292fbc3000 ---p 00172000 09:02 27395008 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
            7f292fbc3000-7f292fbcd000 r--p 00172000 09:02 27395008 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
            7f292fbcd000-7f292fbcf000 rw-p 0017c000 09:02 27395008 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.21
            7f292fbcf000-7f292fbd3000 rw-p 00000000 00:00 0
            7f292fbd3000-7f292fbd4000 r-xp 00000000 09:02 19926412 /lib/x86_64-linux-gnu/libaio.so.1.0.1
            7f292fbd4000-7f292fdd3000 ---p 00001000 09:02 19926412 /lib/x86_64-linux-gnu/libaio.so.1.0.1
            7f292fdd3000-7f292fdd4000 r--p 00000000 09:02 19926412 /lib/x86_64-linux-gnu/libaio.so.1.0.1
            7f292fdd4000-7f292fdd5000 rw-p 00001000 09:02 19926412 /lib/x86_64-linux-gnu/libaio.so.1.0.1
            7f292fdd5000-7f292fdee000 r-xp 00000000 09:02 19922961 /lib/x86_64-linux-gnu/libz.so.1.2.8
            7f292fdee000-7f292ffed000 ---p 00019000 09:02 19922961 /lib/x86_64-linux-gnu/libz.so.1.2.8
            7f292ffed000-7f292ffee000 r--p 00018000 09:02 19922961 /lib/x86_64-linux-gnu/libz.so.1.2.8
            7f292ffee000-7f292ffef000 rw-p 00019000 09:02 19922961 /lib/x86_64-linux-gnu/libz.so.1.2.8
            7f292ffef000-7f292fff8000 r-xp 00000000 09:02 19923170 /lib/x86_64-linux-gnu/libcrypt-2.23.so
            7f292fff8000-7f29301f7000 ---p 00009000 09:02 19923170 /lib/x86_64-linux-gnu/libcrypt-2.23.so
            7f29301f7000-7f29301f8000 r--p 00008000 09:02 19923170 /lib/x86_64-linux-gnu/libcrypt-2.23.so
            7f29301f8000-7f29301f9000 rw-p 00009000 09:02 19923170 /lib/x86_64-linux-gnu/libcrypt-2.23.so
            7f29301f9000-7f2930227000 rw-p 00000000 00:00 0
            7f2930227000-7f293022a000 r-xp 00000000 09:02 19923153 /lib/x86_64-linux-gnu/libdl-2.23.so
            7f293022a000-7f2930429000 ---p 00003000 09:02 19923153 /lib/x86_64-linux-gnu/libdl-2.23.so
            7f2930429000-7f293042a000 r--p 00002000 09:02 19923153 /lib/x86_64-linux-gnu/libdl-2.23.so
            7f293042a000-7f293042b000 rw-p 00003000 09:02 19923153 /lib/x86_64-linux-gnu/libdl-2.23.so
            7f293042b000-7f2930645000 r-xp 00000000 09:02 19922986 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
            7f2930645000-7f2930844000 ---p 0021a000 09:02 19922986 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
            7f2930844000-7f2930860000 r--p 00219000 09:02 19922986 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
            7f2930860000-7f293086c000 rw-p 00235000 09:02 19922986 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0
            7f293086c000-7f293086f000 rw-p 00000000 00:00 0
            7f293086f000-7f29308cd000 r-xp 00000000 09:02 19922987 /lib/x86_64-linux-gnu/libssl.so.1.0.0
            7f29308cd000-7f2930acd000 ---p 0005e000 09:02 19922987 /lib/x86_64-linux-gnu/libssl.so.1.0.0
            7f2930acd000-7f2930ad1000 r--p 0005e000 09:02 19922987 /lib/x86_64-linux-gnu/libssl.so.1.0.0
            7f2930ad1000-7f2930ad8000 rw-p 00062000 09:02 19922987 /lib/x86_64-linux-gnu/libssl.so.1.0.0
            7f2930ad8000-7f2930adf000 r-xp 00000000 09:02 19923169 /lib/x86_64-linux-gnu/librt-2.23.so
            7f2930adf000-7f2930cde000 ---p 00007000 09:02 19923169 /lib/x86_64-linux-gnu/librt-2.23.so
            7f2930cde000-7f2930cdf000 r--p 00006000 09:02 19923169 /lib/x86_64-linux-gnu/librt-2.23.so
            7f2930cdf000-7f2930ce0000 rw-p 00007000 09:02 19923169 /lib/x86_64-linux-gnu/librt-2.23.so
            7f2930ce0000-7f2930db7000 r-xp 00000000 09:02 19922984 /lib/x86_64-linux-gnu/libgcrypt.so.20.0.5
            7f2930db7000-7f2930fb7000 ---p 000d7000 09:02 19922984 /lib/x86_64-linux-gnu/libgcrypt.so.20.0.5
            7f2930fb7000-7f2930fb8000 r--p 000d7000 09:02 19922984 /lib/x86_64-linux-gnu/libgcrypt.so.20.0.5
            7f2930fb8000-7f2930fc0000 rw-p 000d8000 09:02 19922984 /lib/x86_64-linux-gnu/libgcrypt.so.20.0.5
            7f2930fc0000-7f2930fc1000 rw-p 00000000 00:00 0
            7f2930fc1000-7f2930fd9000 r-xp 00000000 09:02 19923150 /lib/x86_64-linux-gnu/libpthread-2.23.so
            7f2930fd9000-7f29311d8000 ---p 00018000 09:02 19923150 /lib/x86_64-linux-gnu/libpthread-2.23.so
            7f29311d8000-7f29311d9000 r--p 00017000 09:02 19923150 /lib/x86_64-linux-gnu/libpthread-2.23.so
            7f29311d9000-7f29311da000 rw-p 00018000 09:02 19923150 /lib/x86_64-linux-gnu/libpthread-2.23.so
            7f29311da000-7f29311de000 rw-p 00000000 00:00 0
            7f29311de000-7f2931204000 r-xp 00000000 09:02 19923075 /lib/x86_64-linux-gnu/ld-2.23.so
            7f2931208000-7f29313fd000 rw-p 00000000 00:00 0
            7f2931402000-7f2931403000 rw-p 00000000 00:00 0
            7f2931403000-7f2931404000 r--p 00025000 09:02 19923075 /lib/x86_64-linux-gnu/ld-2.23.so
            7f2931404000-7f2931405000 rw-p 00026000 09:02 19923075 /lib/x86_64-linux-gnu/ld-2.23.so
            7f2931405000-7f2931406000 rw-p 00000000 00:00 0
            7ffc5fa39000-7ffc5fa5a000 rw-p 00000000 00:00 0 [stack]
            7ffc5fab0000-7ffc5fab3000 r--p 00000000 00:00 0 [vvar]
            7ffc5fab3000-7ffc5fab5000 r-xp 00000000 00:00 0 [vdso]
            ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
            11:38:30 UTC - xtrabackup got signal 6 ;
            This could be because you hit a bug or data is corrupted.
            This error can also be caused by malfunctioning hardware.
            Attempting to collect some information that could help diagnose the problem.
            As this is a crash and something is definitely wrong, the information
            collection process might fail.

      Thread pointer: 0x0
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0 thread_stack 0x10000
      xtrabackup(my_print_stacktrace+0x3b)[0xc40b9b]
      xtrabackup(handle_fatal_signal+0x281)[0xae2761]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f2930fd2390]
      /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x38)[0x7f292ef9d428]
      /lib/x86_64-linux-gnu/libc.so.6(abort+0x16a)[0x7f292ef9f02a]
      /lib/x86_64-linux-gnu/libc.so.6(+0x777ea)[0x7f292efdf7ea]
      /lib/x86_64-linux-gnu/libc.so.6(+0x80dfb)[0x7f292efe8dfb]
      /lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7f292efec53c]
      /lib/x86_64-linux-gnu/libc.so.6(+0x3a025)[0x7f292efa2025]
      /lib/x86_64-linux-gnu/libc.so.6(+0x3a045)[0x7f292efa2045]
      xtrabackup[0x6fd4c7]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f2930fc86ba]
      /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f292f06f41d]

      Please report a bug at https://jira.percona.com/projects/PXB

       

       

       

       

       

       

       

      Attachments

        Activity

          People

            Unassigned Unassigned
            ofrias Oscar
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Smart Checklist