Uploaded image for project: 'Percona XtraDB Cluster'
  1. Percona XtraDB Cluster
  2. PXC-3853

SST fails with encrypt=4 (enabled) on Ubuntu focal/20.04 (32, broken pipe)

Details

    • Bug
    • Status: Confirmation
    • Medium
    • Resolution: Unresolved
    • 5.7.35-31.53 (Q3 2021)
    • None
    • None
    • None
    • Yes

    Description

      On Ubuntu 20.04 I noticed this weird behaviour when doing SST between the peers of my testing setup (3 node cluster). With encryption in the  [sst] section enabled, the SST fails. Without encryption, it succeeds. Encryption for IST is disabled, all IST succeed. The cluster is not heavily loaded.

      Not working:

      joiner: encrypt=4, openssl 1.1.1f-1ubuntu2.9, socat 1.7.3
      donor: encrypt=4, openssl 1.1.1f-1ubuntu2.9, socat 1.7.3
      
      2021-11-23T14:44:18.687203Z 0 [Note] WSREP: (d47c3f47, 'tcp://0.0.0.0:4567') turning message relay requesting off
      2021-11-23T14:44:18.687203Z 0 [Note] WSREP: (d47c3f47, 'tcp://0.0.0.0:4567') turning message relay requesting off 
      2021-11-23T14:45:56.779969Z WSREP_SST: [INFO] Trying to terminate (184143) socat -u TCP-LISTEN:4444,reuseaddr,retry=30 stdio | xbstream -x  $xbstream_eopts with SIGTERM 
      2021-11-23T14:45:57.789677Z WSREP_SST: [ERROR] ******************* FATAL ERROR **********************  
      2021-11-23T14:45:57.791902Z WSREP_SST: [ERROR] Possible timeout in receving first data from donor in gtid/keyring stage 
      2021-11-23T14:45:57.793610Z WSREP_SST: [ERROR] ******************************************************  
      2021-11-23T14:45:57.798276Z WSREP_SST: [ERROR] Cleanup after exit with status:32
      2021-11-23T14:45:57.813580Z 0 [ERROR] WSREP: Process completed with error: wsrep_sst_xtrabackup-v2 --role 'joiner' --address '10.0.0.1' --datadir '/data/mysql/' --defaults-file '/etc/mysql/my.cnf' --defaults-group-suffix '' --parent '183695' --mysqld-version '5.7.35-38-57'   '' : 32 (Broken pipe)
      2021-11-23T14:45:57.813641Z 0 [ERROR] WSREP: Failed to read uuid:seqno from joiner script.
      2021-11-23T14:45:57.813650Z 0 [ERROR] WSREP: SST script aborted with error 32 (Broken pipe)
      2021-11-23T14:45:57.813733Z 0 [ERROR] WSREP: SST failed: 32 (Broken pipe)
      2021-11-23T14:45:57.813755Z 0 [ERROR] Aborting

       

      Working:

      joiner: encrypt=0
      donor: encrypt=0
      2021-11-23T16:22:37.672094Z WSREP_SST: [INFO] Proceeding with SST......... 
      2021-11-23T16:22:37.789409Z WSREP_SST: [INFO] ............Waiting for SST streaming to complete!
      2021-11-23T16:22:38.676787Z 0 [Note] WSREP: (912784da, 'tcp://0.0.0.0:4567') turning message relay requesting off
      2021-11-23T16:25:43.833719Z 0 [Note] WSREP: 0.0 (pxcdb01a): State transfer to 2.0 (pxcdb01b) complete.
      2021-11-23T16:25:43.837885Z 0 [Note] WSREP: Member 0.0 (pxcdb01a) synced with group. 
      2021-11-23T16:25:50.101898Z WSREP_SST: [INFO] Preparing the backup at /data/mysql//.sst 
      2021-11-23T16:26:01.306355Z WSREP_SST: [INFO] Moving the backup to /data/mysql/ 
      2021-11-23T16:26:01.426161Z WSREP_SST: [INFO] Galera co-ords from recovery: 4c6d5f6a-0083-11ea-b5eb-07961329772c:67751838
      2021-11-23T16:26:01.449340Z 0 [Note] WSREP: SST complete, seqno: 67751838
      

      Config is done in my.cnf:

      [sst]
      encrypt=4
      ssl-ca = /etc/ssl/certs/ca-certificates.crt
      ssl-cert = /etc/directory/ssl/internal-web.crt
      ssl-key = /etc/directory/ssl/internal-web.key
      

      I sincerly hope this is not a config mistake, if it is, please let me know.

      Attachments

        Activity

          People

            aaditya.dubey Aaditya Dubey
            masc Malte Schmidt
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:

              Smart Checklist