Uploaded image for project: 'Percona XtraDB Cluster'
  1. Percona XtraDB Cluster
  2. PXC-3691

garbd doesn't correctly handle multiple incoming SST requests at the same time

Details

    • Bug
    • Status: Pending Release
    • Medium
    • Resolution: Done
    • 5.7.32-31.47
    • 5.7.38-31.59 (Q2 2022)
    • None
    • Yes

    Description

      Hi team,

      While I was investigating the issue K8SPXC-782 which was created by our support team I found and reproduced that with PXC 5.7 we have exactly the same issue which was described and fixed under the task https://jira.percona.com/browse/PXC-3301 for PXC 8.0.

      From PXC log:

      2021-06-15T14:43:55.355256Z 0 [Note] WSREP: Member 0.0 (garb) requested state transfer from 'scheduled-backup-pxc-2'. Selected 4.0 (scheduled-backup-pxc-2)(SYNCED) as donor.
      2021-06-15T14:43:55.355276Z 0 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 29)
      2021-06-15T14:43:55.355299Z 0 [Warning] WSREP: Member 1.0 (garb) requested state transfer from 'scheduled-backup-pxc-2', but it is impossible to select State Transfer donor: Resource temporarily unavailable
      

      As you can see from the log we have the situation that member 1b3135b7 was rejected and it was handled correctly by backup script but member 1b44005b was not correctly processed at all and as a result backup hangs and customer has backup which is never ended.

      You can find all logs in the attachment . 

      P.S. I understand that it is 5.7 and from our end at least it is enough to have some error message in garbd log which we can 'get' and understand that something goes wrong with backup.

      Attachments

        Issue Links

          Activity

            People

              amonar Anton Matvienko
              slava.sarzhan Slava Sarzhan
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - Not Specified
                  Not Specified
                  Logged:
                  Time Spent - 30 minutes
                  30m

                  Smart Checklist