Uploaded image for project: 'Percona Operator for MySQL based on Percona XtraDB Cluster'
  1. Percona Operator for MySQL based on Percona XtraDB Cluster
  2. K8SPXC-937

[WSREP-SST] 2022/01/11 18:06:20 socat[23163] E connect(7, AF=2 10.233.86.195:4444, 16): Connection timed out"}]

Details

    • Bug
    • Status: New
    • Low
    • Resolution: Unresolved
    • 1.7.0
    • None
    • Community Activities
    • Yes

    Description

      First of all, I just wanted to report an issue for your awareness. Not expecting to fix it as I already solved it. May help others in the future too.

      If you have any bet on what was happened that would be awesome, otherwise please feel free to close it.


      Okay, I have a working PXC operator for a year with 3 nodes and experienced an issue recently.

      It could be related with pxc-backup.


      Here is what I did:

      1) I removed a disk from MinIO. So this service was left running, but obviously not working for some time. It was not possible to push backups to it. This was expected.

      2) After some hours XtraDB cluster failed with the errors attached.

      I had to do a crash recovery as described in https://www.percona.com/doc/kubernetes-operator-for-pxc/recovery.html

      Recovered successfully.

      3) After several hours issue repeated


      The issue lasted until I removed this block from pxc:

        backup:
          enabled: true
          image: percona/percona-xtradb-cluster-operator:1.7.0-pxc8.0-backup
          pitr:
            enabled: false
            storageName: STORAGE-NAME-HERE
            timeBetweenUploads: 60
          storages:
            minio-2:
              s3:
                bucket: percona
                credentialsSecret: db-backup-s3
                endpointUrl: http://192.168.240.100:9000
                region: us-east-1
              type: s3
          schedule:
          - keep: 24
            name: minio-backup-2
            schedule: '0 8,11,14,17,19 * * *'
            storageName: minio-2
          - keep: 30
            name: minio-backup-daily-2
            schedule: 0 3 * * *
            storageName: minio-2
          resources:
            requests:
              memory: 300M
              cpu: 80m
            limits:
              memory: 400M
              cpu: 200m
      
      

      pxc-0

      [0] db-p.db-cluster-pxc-0.mysqld-error.log: [1641924097.398457225, {"log"=>"2022-01-11T18:01:37.398319Z 0 [Warning] [MY-000000] [Galera] Could not find peer: f70d35df-72ff-11ec-b7c8-2a60f8408881"}]
      [1] db-p.db-cluster-pxc-0.mysqld-error.log: [1641924097.398460760, {"log"=>"2022-01-11T18:01:37.398433Z 0 [Warning] [MY-000000] [Galera] 1.0 (db-cluster-pxc-2): State transfer to -1.-1 (left the group) failed: -32 (Broken pipe)"}]
      [2] db-p.db-cluster-pxc-0.mysqld-error.log: [1641924097.406815231, {"log"=>"2022-01-11T18:01:37.406655Z 0 [Note] [MY-000000] [Galera] Member 1.0 (db-cluster-pxc-2) synced with group."}]
      [0] db-p.db-cluster-pxc-0.mysqld-error.log: [1641924368.749930875, {"log"=>"2022-01-11T18:06:08.749805Z 0 [Warning] [MY-000000] [Galera] Could not find peer: fde09f94-72ff-11ec-905d-62079652935c"}]
      [1] db-p.db-cluster-pxc-0.mysqld-error.log: [1641924368.749934058, {"log"=>"2022-01-11T18:06:08.749880Z 0 [Warning] [MY-000000] [Galera] 0.0 (db-cluster-pxc-1): State transfer to -1.-1 (left the group) failed: -32 (Broken pipe)"}]
      [2] db-p.db-cluster-pxc-0.mysqld-error.log: [1641924368.762522524, {"log"=>"2022-01-11T18:06:08.762425Z 0 [Note] [MY-000000] [Galera] Member 0.0 (db-cluster-pxc-1) synced with group."}]
      [0] db-p.db-cluster-pxc-0.mysqld-error.log: [1641924380.263124604, {"log"=>"2022-01-11T18:06:20.262963Z 0 [Note] [MY-000000] [WSREP-SST] 2022/01/11 18:06:20 socat[23163] E connect(7, AF=2 10.233.86.195:4444, 16): Connection timed out"}]
      [1] db-p.db-cluster-pxc-0.mysqld-error.log: [1641924380.263793581, {"log"=>"2022-01-11T18:06:20.263599Z 0 [ERROR] [MY-000000] [WSREP-SST] ******************* FATAL ERROR ********************** "}]
      [2] db-p.db-cluster-pxc-0.mysqld-error.log: [1641924380.263794935, {"log"=>"2022-01-11T18:06:20.263708Z 0 [ERROR] [MY-000000] [WSREP-SST] Error while sending data to joiner node:  exit codes: 0 1"}]
      [3] db-p.db-cluster-pxc-0.mysqld-error.log: [1641924380.263795245, {"log"=>"2022-01-11T18:06:20.263722Z 0 [ERROR] [MY-000000] [WSREP-SST] Line 1296"}]
      [4] db-p.db-cluster-pxc-0.mysqld-error.log: [1641924380.263795539, {"log"=>"2022-01-11T18:06:20.263755Z 0 [ERROR] [MY-000000] [WSREP-SST] ****************************************************** "}]
      [5] db-p.db-cluster-pxc-0.mysqld-error.log: [1641924380.264664175, {"log"=>"2022-01-11T18:06:20.264019Z 0 [ERROR] [MY-000000] [WSREP-SST] Cleanup after exit with status:32"}]
      [6] db-p.db-cluster-pxc-0.mysqld-error.log: [1641924380.284579349, {"log"=>"2022-01-11T18:06:20.276768Z 0 [ERROR] [MY-000000] [WSREP] Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '10.233.86.195:4444/xtrabackup_sst//1' --socket '/tmp/mysql.sock' --datadir '/var/lib/mysql/' --basedir '/usr/' --plugindir '/usr/lib64/mysql/plugin/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --mysqld-version '8.0.21-12.1'  --binlog 'binlog' --gtid '9bde390e-959d-11eb-9e1a-2692c965adf8:750280' : 32 (Broken pipe)"}]
      [7] db-p.db-cluster-pxc-0.mysqld-error.log: [1641924380.286411341, {"log"=>"2022-01-11T18:06:20.286301Z 0 [Note] [MY-000000] [Galera] SST sending failed: -32"}]
      [8] db-p.db-cluster-pxc-0.mysqld-error.log: [1641924380.286413354, {"log"=>"2022-01-11T18:06:20.286351Z 0 [Note] [MY-000000] [WSREP] Server status change donor -> joined"}]
      [9] db-p.db-cluster-pxc-0.mysqld-error.log: [1641924380.286413659, {"log"=>"2022-01-11T18:06:20.286384Z 0 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification."}]
      [10] db-p.db-cluster-pxc-0.mysqld-error.log: [1641924380.286773476, {"log"=>"2022-01-11T18:06:20.286675Z 0 [ERROR] [MY-000000] [WSREP] Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '10.233.86.195:4444/xtrabackup_sst//1' --socket '/tmp/mysql.sock' --datadir '/var/lib/mysql/' --basedir '/usr/' --plugindir '/usr/lib64/mysql/plugin/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --mysqld-version '8.0.21-12.1'  --binlog 'binlog' --gtid '9bde390e-959d-11eb-9e1a-2692c965adf8:750280' "}]
      [11] db-p.db-cluster-pxc-0.mysqld-error.log: [1641924380.298424411, {"log"=>"2022-01-11T18:06:20.298271Z 0 [Warning] [MY-000000] [Galera] Could not find peer: 041c72f6-7300-11ec-bec4-d3dcea7f4868"}]
      [12] db-p.db-cluster-pxc-0.mysqld-error.log: [1641924380.298427079, {"log"=>"2022-01-11T18:06:20.298358Z 0 [Warning] [MY-000000] [Galera] 2.0 (db-cluster-pxc-0): State transfer to -1.-1 (left the group) failed: -32 (Broken pipe)"}]
      [13] db-p.db-cluster-pxc-0.mysqld-error.log: [1641924380.298427464, {"log"=>"2022-01-11T18:06:20.298377Z 0 [Note] [MY-000000] [Galera] Shifting DONOR/DESYNCED -> JOINED (TO: 750301)"}]
      [14] db-p.db-cluster-pxc-0.mysqld-error.log: [1641924380.309189924, {"log"=>"2022-01-11T18:06:20.309043Z 0 [Note] [MY-000000] [Galera] Member 2.0 (db-cluster-pxc-0) synced with group."}]
      [15] db-p.db-cluster-pxc-0.mysqld-error.log: [1641924380.309192935, {"log"=>"2022-01-11T18:06:20.309094Z 0 [Note] [MY-000000] [Galera] Shifting JOINED -> SYNCED (TO: 750301)"}]
      [16] db-p.db-cluster-pxc-0.mysqld-error.log: [1641924380.309250287, {"log"=>"2022-01-11T18:06:20.309202Z 10 [Note] [MY-000000] [Galera] Server db-cluster-pxc-0 synced with group"}]
      [17] db-p.db-cluster-pxc-0.mysqld-error.log: [1641924380.309264124, {"log"=>"2022-01-11T18:06:20.309252Z 10 [Note] [MY-000000] [WSREP] Server status change joined -> synced"}]
      [18] db-p.db-cluster-pxc-0.mysqld-error.log: [1641924380.309280817, {"log"=>"2022-01-11T18:06:20.309267Z 10 [Note] [MY-000000] [WSREP] Synchronized with group, ready for connections"}]
      [19] db-p.db-cluster-pxc-0.mysqld-error.log: [1641924380.309381695, {"log"=>"2022-01-11T18:06:20.309332Z 10 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification."}]
      $ ~/Documents/git $ k get pods -o wide | grep 10.233.86.195
      xb-cron-db-cluster-minio-2-20220111170015-fa5m3-6qzhk   0/1     Error       0          79m     10.233.86.195    node-2.internal   <none>           <none>
      $ ~/Documents/git $
      

      pxc-1

      [0] db-p.db-cluster-pxc-1.mysqld-error.log: [1641924097.397582985, {"log"=>"2022-01-11T18:01:37.397381Z 0 [Warning] [MY-000000] [Galera] Could not find peer: f70d35df-72ff-11ec-b7c8-2a60f8408881"}]
      [1] db-p.db-cluster-pxc-1.mysqld-error.log: [1641924097.397586157, {"log"=>"2022-01-11T18:01:37.397532Z 0 [Warning] [MY-000000] [Galera] 1.0 (db-cluster-pxc-2): State transfer to -1.-1 (left the group) failed: -32 (Broken pipe)"}]
      [2] db-p.db-cluster-pxc-1.mysqld-error.log: [1641924097.407972873, {"log"=>"2022-01-11T18:01:37.407823Z 0 [Note] [MY-000000] [Galera] Member 1.0 (db-cluster-pxc-2) synced with group."}]
      [0] db-p.db-cluster-pxc-1.mysqld-error.log: [1641924368.721775962, {"log"=>"2022-01-11T18:06:08.721620Z 0 [Note] [MY-000000] [WSREP-SST] 2022/01/11 18:06:08 socat[11743] E connect(7, AF=2 10.233.86.43:4444, 16): Connection timed out"}]
      [1] db-p.db-cluster-pxc-1.mysqld-error.log: [1641924368.723277609, {"log"=>"2022-01-11T18:06:08.722721Z 0 [ERROR] [MY-000000] [WSREP-SST] ******************* FATAL ERROR ********************** "}]
      [2] db-p.db-cluster-pxc-1.mysqld-error.log: [1641924368.723279133, {"log"=>"2022-01-11T18:06:08.722789Z 0 [ERROR] [MY-000000] [WSREP-SST] Error while sending data to joiner node:  exit codes: 0 1"}]
      [3] db-p.db-cluster-pxc-1.mysqld-error.log: [1641924368.723279562, {"log"=>"2022-01-11T18:06:08.722805Z 0 [ERROR] [MY-000000] [WSREP-SST] Line 1296"}]
      [4] db-p.db-cluster-pxc-1.mysqld-error.log: [1641924368.723279896, {"log"=>"2022-01-11T18:06:08.722840Z 0 [ERROR] [MY-000000] [WSREP-SST] ****************************************************** "}]
      [5] db-p.db-cluster-pxc-1.mysqld-error.log: [1641924368.723280238, {"log"=>"2022-01-11T18:06:08.722979Z 0 [ERROR] [MY-000000] [WSREP-SST] Cleanup after exit with status:32"}]
      [6] db-p.db-cluster-pxc-1.mysqld-error.log: [1641924368.734651033, {"log"=>"2022-01-11T18:06:08.734532Z 0 [ERROR] [MY-000000] [WSREP] Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '10.233.86.43:4444/xtrabackup_sst//1' --socket '/tmp/mysql.sock' --datadir '/var/lib/mysql/' --basedir '/usr/' --plugindir '/usr/lib64/mysql/plugin/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --mysqld-version '8.0.21-12.1'  --binlog 'binlog' --gtid '9bde390e-959d-11eb-9e1a-2692c965adf8:750277' : 32 (Broken pipe)"}]
      [7] db-p.db-cluster-pxc-1.mysqld-error.log: [1641924368.736546594, {"log"=>"2022-01-11T18:06:08.736344Z 0 [Note] [MY-000000] [Galera] SST sending failed: -32"}]
      [8] db-p.db-cluster-pxc-1.mysqld-error.log: [1641924368.736561514, {"log"=>"2022-01-11T18:06:08.736413Z 0 [Note] [MY-000000] [WSREP] Server status change donor -> joined"}]
      [9] db-p.db-cluster-pxc-1.mysqld-error.log: [1641924368.736561903, {"log"=>"2022-01-11T18:06:08.736487Z 0 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification."}]
      [10] db-p.db-cluster-pxc-1.mysqld-error.log: [1641924368.736587657, {"log"=>"2022-01-11T18:06:08.736547Z 0 [ERROR] [MY-000000] [WSREP] Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '10.233.86.43:4444/xtrabackup_sst//1' --socket '/tmp/mysql.sock' --datadir '/var/lib/mysql/' --basedir '/usr/' --plugindir '/usr/lib64/mysql/plugin/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --mysqld-version '8.0.21-12.1'  --binlog 'binlog' --gtid '9bde390e-959d-11eb-9e1a-2692c965adf8:750277' "}]
      [11] db-p.db-cluster-pxc-1.mysqld-error.log: [1641924368.747247838, {"log"=>"2022-01-11T18:06:08.747111Z 0 [Warning] [MY-000000] [Galera] Could not find peer: fde09f94-72ff-11ec-905d-62079652935c"}]
      [12] db-p.db-cluster-pxc-1.mysqld-error.log: [1641924368.747250462, {"log"=>"2022-01-11T18:06:08.747161Z 0 [Warning] [MY-000000] [Galera] 0.0 (db-cluster-pxc-1): State transfer to -1.-1 (left the group) failed: -32 (Broken pipe)"}]
      [13] db-p.db-cluster-pxc-1.mysqld-error.log: [1641924368.747250859, {"log"=>"2022-01-11T18:06:08.747187Z 0 [Note] [MY-000000] [Galera] Shifting DONOR/DESYNCED -> JOINED (TO: 750300)"}]
      [14] db-p.db-cluster-pxc-1.mysqld-error.log: [1641924368.763045407, {"log"=>"2022-01-11T18:06:08.762914Z 0 [Note] [MY-000000] [Galera] Member 0.0 (db-cluster-pxc-1) synced with group."}]
      [15] db-p.db-cluster-pxc-1.mysqld-error.log: [1641924368.763048597, {"log"=>"2022-01-11T18:06:08.762977Z 0 [Note] [MY-000000] [Galera] Shifting JOINED -> SYNCED (TO: 750300)"}]
      [16] db-p.db-cluster-pxc-1.mysqld-error.log: [1641924368.763076762, {"log"=>"2022-01-11T18:06:08.763027Z 10 [Note] [MY-000000] [Galera] Server db-cluster-pxc-1 synced with group"}]
      [17] db-p.db-cluster-pxc-1.mysqld-error.log: [1641924368.763077274, {"log"=>"2022-01-11T18:06:08.763049Z 10 [Note] [MY-000000] [WSREP] Server status change joined -> synced"}]
      [18] db-p.db-cluster-pxc-1.mysqld-error.log: [1641924368.763077591, {"log"=>"2022-01-11T18:06:08.763061Z 10 [Note] [MY-000000] [WSREP] Synchronized with group, ready for connections"}]
      [19] db-p.db-cluster-pxc-1.mysqld-error.log: [1641924368.763090258, {"log"=>"2022-01-11T18:06:08.763072Z 10 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification."}]
      [0] db-p.db-cluster-pxc-1.mysqld-error.log: [1641924380.295562866, {"log"=>"2022-01-11T18:06:20.295300Z 0 [Warning] [MY-000000] [Galera] Could not find peer: 041c72f6-7300-11ec-bec4-d3dcea7f4868"}]
      [1] db-p.db-cluster-pxc-1.mysqld-error.log: [1641924380.295566537, {"log"=>"2022-01-11T18:06:20.295397Z 0 [Warning] [MY-000000] [Galera] 2.0 (db-cluster-pxc-0): State transfer to -1.-1 (left the group) failed: -32 (Broken pipe)"}]
      [2] db-p.db-cluster-pxc-1.mysqld-error.log: [1641924380.308288501, {"log"=>"2022-01-11T18:06:20.308156Z 0 [Note] [MY-000000] [Galera] Member 2.0 (db-cluster-pxc-0) synced with group."}]
      $ ~/Documents/git $ k get pods -o wide | grep 10.233.86.43
      xb-cron-db-cluster-minio-2-20220111170015-fa5m3-xw75t   0/1     Error       0          80m     10.233.86.43     node-2.internal   <none>           <none>
      $ ~/Documents/git $
      

      pxc-2

      [0] db-p.db-cluster-pxc-2.mysqld-error.log: [1641924097.352724221, {"log"=>"2022-01-11T18:01:37.352489Z 0 [Note] [MY-000000] [WSREP-SST] 2022/01/11 18:01:37 socat[11943] E connect(7, AF=2 10.233.86.85:4444, 16): Connection timed out"}]
      [1] db-p.db-cluster-pxc-2.mysqld-error.log: [1641924097.352727724, {"log"=>"2022-01-11T18:01:37.352558Z 0 [ERROR] [MY-000000] [WSREP-SST] ******************* FATAL ERROR ********************** "}]
      [2] db-p.db-cluster-pxc-2.mysqld-error.log: [1641924097.352728057, {"log"=>"2022-01-11T18:01:37.352580Z 0 [ERROR] [MY-000000] [WSREP-SST] Error while sending data to joiner node:  exit codes: 0 1"}]
      [3] db-p.db-cluster-pxc-2.mysqld-error.log: [1641924097.352728322, {"log"=>"2022-01-11T18:01:37.352590Z 0 [ERROR] [MY-000000] [WSREP-SST] Line 1296"}]
      [4] db-p.db-cluster-pxc-2.mysqld-error.log: [1641924097.352728567, {"log"=>"2022-01-11T18:01:37.352598Z 0 [ERROR] [MY-000000] [WSREP-SST] ****************************************************** "}]
      [5] db-p.db-cluster-pxc-2.mysqld-error.log: [1641924097.352728814, {"log"=>"2022-01-11T18:01:37.352606Z 0 [ERROR] [MY-000000] [WSREP-SST] Cleanup after exit with status:32"}]
      [6] db-p.db-cluster-pxc-2.mysqld-error.log: [1641924097.370959841, {"log"=>"2022-01-11T18:01:37.370756Z 0 [ERROR] [MY-000000] [WSREP] Process completed with error: wsrep_sst_xtrabackup-v2 --role 'donor' --address '10.233.86.85:4444/xtrabackup_sst//1' --socket '/tmp/mysql.sock' --datadir '/var/lib/mysql/' --basedir '/usr/' --plugindir '/usr/lib64/mysql/plugin/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --mysqld-version '8.0.21-12.1'  --binlog 'binlog' --gtid '9bde390e-959d-11eb-9e1a-2692c965adf8:750275' : 32 (Broken pipe)"}]
      [7] db-p.db-cluster-pxc-2.mysqld-error.log: [1641924097.372589022, {"log"=>"2022-01-11T18:01:37.372466Z 0 [Note] [MY-000000] [Galera] SST sending failed: -32"}]
      [8] db-p.db-cluster-pxc-2.mysqld-error.log: [1641924097.372591781, {"log"=>"2022-01-11T18:01:37.372511Z 0 [Note] [MY-000000] [WSREP] Server status change donor -> joined"}]
      [9] db-p.db-cluster-pxc-2.mysqld-error.log: [1641924097.372592078, {"log"=>"2022-01-11T18:01:37.372565Z 0 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification."}]
      [10] db-p.db-cluster-pxc-2.mysqld-error.log: [1641924097.373085260, {"log"=>"2022-01-11T18:01:37.372914Z 0 [ERROR] [MY-000000] [WSREP] Command did not run: wsrep_sst_xtrabackup-v2 --role 'donor' --address '10.233.86.85:4444/xtrabackup_sst//1' --socket '/tmp/mysql.sock' --datadir '/var/lib/mysql/' --basedir '/usr/' --plugindir '/usr/lib64/mysql/plugin/' --defaults-file '/etc/my.cnf' --defaults-group-suffix '' --mysqld-version '8.0.21-12.1'  --binlog 'binlog' --gtid '9bde390e-959d-11eb-9e1a-2692c965adf8:750275' "}]
      [11] db-p.db-cluster-pxc-2.mysqld-error.log: [1641924097.390596549, {"log"=>"2022-01-11T18:01:37.390427Z 0 [Warning] [MY-000000] [Galera] Could not find peer: f70d35df-72ff-11ec-b7c8-2a60f8408881"}]
      [12] db-p.db-cluster-pxc-2.mysqld-error.log: [1641924097.390599933, {"log"=>"2022-01-11T18:01:37.390481Z 0 [Warning] [MY-000000] [Galera] 1.0 (db-cluster-pxc-2): State transfer to -1.-1 (left the group) failed: -32 (Broken pipe)"}]
      [13] db-p.db-cluster-pxc-2.mysqld-error.log: [1641924097.390600264, {"log"=>"2022-01-11T18:01:37.390505Z 0 [Note] [MY-000000] [Galera] Shifting DONOR/DESYNCED -> JOINED (TO: 750281)"}]
      [14] db-p.db-cluster-pxc-2.mysqld-error.log: [1641924097.400481070, {"log"=>"2022-01-11T18:01:37.400364Z 0 [Note] [MY-000000] [Galera] Member 1.0 (db-cluster-pxc-2) synced with group."}]
      [15] db-p.db-cluster-pxc-2.mysqld-error.log: [1641924097.400484296, {"log"=>"2022-01-11T18:01:37.400410Z 0 [Note] [MY-000000] [Galera] Shifting JOINED -> SYNCED (TO: 750281)"}]
      [16] db-p.db-cluster-pxc-2.mysqld-error.log: [1641924097.400511197, {"log"=>"2022-01-11T18:01:37.400481Z 1 [Note] [MY-000000] [Galera] Server db-cluster-pxc-2 synced with group"}]
      [17] db-p.db-cluster-pxc-2.mysqld-error.log: [1641924097.400521258, {"log"=>"2022-01-11T18:01:37.400512Z 1 [Note] [MY-000000] [WSREP] Server status change joined -> synced"}]
      [18] db-p.db-cluster-pxc-2.mysqld-error.log: [1641924097.400534875, {"log"=>"2022-01-11T18:01:37.400524Z 1 [Note] [MY-000000] [WSREP] Synchronized with group, ready for connections"}]
      [19] db-p.db-cluster-pxc-2.mysqld-error.log: [1641924097.400543887, {"log"=>"2022-01-11T18:01:37.400534Z 1 [Note] [MY-000000] [WSREP] wsrep_notify_cmd is not defined, skipping notification."}]
      [0] db-p.db-cluster-pxc-2.mysqld-error.log: [1641924368.748245296, {"log"=>"2022-01-11T18:06:08.748061Z 0 [Warning] [MY-000000] [Galera] Could not find peer: fde09f94-72ff-11ec-905d-62079652935c"}]
      [1] db-p.db-cluster-pxc-2.mysqld-error.log: [1641924368.748248629, {"log"=>"2022-01-11T18:06:08.748120Z 0 [Warning] [MY-000000] [Galera] 0.0 (db-cluster-pxc-1): State transfer to -1.-1 (left the group) failed: -32 (Broken pipe)"}]
      [2] db-p.db-cluster-pxc-2.mysqld-error.log: [1641924368.763576717, {"log"=>"2022-01-11T18:06:08.763443Z 0 [Note] [MY-000000] [Galera] Member 0.0 (db-cluster-pxc-1) synced with group."}]
      [0] db-p.db-cluster-pxc-2.mysqld-error.log: [1641924380.296875310, {"log"=>"2022-01-11T18:06:20.296600Z 0 [Warning] [MY-000000] [Galera] Could not find peer: 041c72f6-7300-11ec-bec4-d3dcea7f4868"}]
      [1] db-p.db-cluster-pxc-2.mysqld-error.log: [1641924380.296878643, {"log"=>"2022-01-11T18:06:20.296663Z 0 [Warning] [MY-000000] [Galera] 2.0 (db-cluster-pxc-0): State transfer to -1.-1 (left the group) failed: -32 (Broken pipe)"}]
      [2] db-p.db-cluster-pxc-2.mysqld-error.log: [1641924380.308418261, {"log"=>"2022-01-11T18:06:20.308135Z 0 [Note] [MY-000000] [Galera] Member 2.0 (db-cluster-pxc-0) synced with group."}]
      $ ~/Documents/git $ k get pods -o wide | grep 10.233.86.85
      xb-cron-db-cluster-minio-2-20220111170015-fa5m3-vjrz7   0/1     Error       0          80m     10.233.86.85     node-2.internal   <none>           <none>
      $ ~/Documents/git $
      

      Attachments

        Activity

          People

            natalia.marukovich Natalia Marukovich
            laimison Laimis V
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:

              Smart Checklist