Uploaded image for project: 'Percona XtraBackup'
  1. Percona XtraBackup
  2. PXB-2625

Error in the HTTP2 framing layer / Large backup fails on Debian Buster

Details

    • Bug
    • Status: Done
    • Medium
    • Resolution: Fixed
    • None
    • 8.0.27-19
    • None
    • None
    • Yes
    • Yes

    Description

      I have a question about Percna Xtrabackup and Debian Buster. I have a server that i recently upgraded to Debian Buster and after that i noticed that the streaming backups to GCP buckets stopped working on it. I then set up another slave that runs Debian Stretch and those backups complete without issues.

      The hardware on those two servers are identical (PowerEdge R640 with NVME storage). They both run Percona Xtrabackup 2.4.24 but the Buster server has NIC driver: mlx5_core version: 5.0-0 and the Stretch server has 3.0-1.

      The Buster server backups fails with this error :

      xtrabackup: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)
      xb_stream_write_data() failed.
      compress: write to the destination stream failed.
      xtrabackup: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)
      [01] xtrabackup: Error: xtrabackup_copy_datafile() failed.
      [01] xtrabackup: Error: failed to copy datafile.
      xtrabackup: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)
      xb_stream_write_data() failed.
      compress: write to the destination stream failed.
      xtrabackup: Error: write to logfile failed
      xtrabackup: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)
      xtrabackup: Error: xtrabackup_copy_logfile() failed.
      

      This is the same type of error we got before version 2.4.24 and the possibility to add the --http-retriable-errors=502 option.

      This error showed right before the backup fails :
      211022 11:21:30 xbcloud: Operation failed. Error: Error in the HTTP2 framing layer

      When running with --verbose, it says this :

      * Expire in 0 ms for 6 (transfer 0x556afda32720)
      211022 12:28:08 xbcloud: successfully uploaded chunk: testbackup2/<database>/<table>.ibd.qp.00000000000000000248, size: 10485840
      < HTTP/2 200 
      < x-guploader-uploadid: ADPycdtlideOqK5O7oWtvWlComgHkKhJq772vCphIXUPjLjLDndmdS7JbJ7p-ltPzy4S1zLBrTnCro4DbmGq3uwQhZeRdKNdRw
      < etag: "06e22ebae21e1db1612621525a793910"
      < x-goog-generation: 1634898488678774
      < x-goog-metageneration: 1
      < x-goog-hash: crc32c=45QALg==
      < x-goog-hash: md5=BuIuuuIeHbFhJiFSWnk5EA==
      < x-goog-stored-content-length: 10485840
      < x-goog-stored-content-encoding: identity
      < vary: Origin
      < content-length: 0
      < date: Fri, 22 Oct 2021 10:28:08 GMT
      < server: UploadServer
      < content-type: text/html; charset=UTF-8
      < 
      * Connection #1 to host ***.storage.googleapis.com left intact
      211022 12:28:08 xbcloud: successfully uploaded chunk: testbackup2/<database>/<table>.ibd.qp.00000000000000000249, size: 10485840
      211022 12:28:08 xbcloud: Upload failed.
      xtrabackup: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)
      xb_stream_write_data() failed.
      compress: write to the destination stream failed.
      xtrabackup: Error writing file 'UNOPENED' (Errcode: 32 - Broken pipe)
      [01] xtrabackup: Error: xtrabackup_copy_datafile() failed.
      [01] xtrabackup: Error: failed to copy datafile.
      211022 12:28:08 >> log scanned up to (104272760567015)
      
      

      Attachments

        1. backup.log.gz
          3.90 MB
        2. backupaa.gz
          1.35 MB
        3. backupab.gz
          1.36 MB
        4. backupac.gz
          1.36 MB
        5. backupad.gz
          1.36 MB
        6. backupae.gz
          1.35 MB
        7. backupaf.gz
          1.34 MB
        8. backupag.gz
          1.36 MB
        9. backupah.gz
          1.36 MB
        10. backupai.gz
          1.37 MB
        11. backupaj.gz
          1.35 MB
        12. backupak.gz
          1.35 MB
        13. backupal.gz
          1.34 MB
        14. backupam.gz
          1.37 MB
        15. backupan.gz
          1.36 MB
        16. backupao.gz
          1.37 MB
        17. backupap.gz
          1.35 MB
        18. backupaq.gz
          1.35 MB
        19. backupar.gz
          1.38 MB
        20. backupas.gz
          1.36 MB
        21. backupat.gz
          1.37 MB
        22. backupau.gz
          1.35 MB
        23. backupav.gz
          1.34 MB
        24. backupaw.gz
          1.35 MB
        25. backupax.gz
          1.36 MB
        26. backupay.gz
          1.35 MB
        27. backupaz.gz
          1.35 MB
        28. backupba.gz
          1.34 MB
        29. backupbb.gz
          1.33 MB
        30. backupbc.gz
          1.36 MB
        31. backupbd.gz
          1.36 MB
        32. backupbe.gz
          1.36 MB
        33. backupbf.gz
          1.35 MB
        34. backupbg.gz
          1.27 MB
        35. backupbh.gz
          729 kB

        Activity

          People

            marcelo.altmann Marcelo Altmann
            Catoman Johan Andersson
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - Not Specified
                Not Specified
                Logged:
                Time Spent - 30 minutes
                30m

                Smart Checklist