Uploaded image for project: 'Percona Backup for MongoDB'
  1. Percona Backup for MongoDB
  2. PBM-485

Fix backups to S3 failing with MaxUploadParts limit by auto-adjusting uploadPartSize value

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Priority: High
    • Resolution: Fixed
    • Affects Version/s: 1.2.0
    • Fix Version/s: 1.3.2
    • Component/s: None
    • Labels:
      None

      Description

      Hi,

      The backup to S3 of our biggest cluster fails with:

       

      Backup with name <backup_name> ended with error. mongodump: write data: upload to S3: MultipartUpload: upload multipart failed     upload id: <upload_id> caused by: TotalPartsExceeded: exceeded total allowed configured MaxUploadParts (10000). Adjust PartSize to fit in this limit.. Please check.
      

      Accordingly to S3 documentation (https://docs.aws.amazon.com/sdk-for-go/api/service/s3/s3manager/), if the UploadPartSize if not defined, it is set to DefaultUploadPartSize: 5MB.

      Is it possible to automatically adjust Part Size on upload? Maybe this should be addressed for the downloads too.

      Thank you.

       

        Smart Checklist

          Attachments

            Issue Links

              Activity

                People

                Assignee:
                andrew.pogrebnoi Andrii Pogrebnoi
                Reporter:
                pedroalb Pedro Albuquerque
                Votes:
                1 Vote for this issue
                Watchers:
                7 Start watching this issue

                  Dates

                  Created:
                  Updated:
                  Resolved:

                    Time Tracking

                    Estimated:
                    Original Estimate - Not Specified
                    Not Specified
                    Remaining:
                    Remaining Estimate - Not Specified
                    Not Specified
                    Logged:
                    Time Spent - 2 days, 4 hours, 20 minutes
                    2d 4h 20m