Uploaded image for project: 'Percona XtraBackup'
  1. Percona XtraBackup
  2. PXB-1813

S3 support in xbcloud

    XMLWordPrintable

    Details

    • Type: New Feature
    • Status: Done
    • Priority: Medium
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.4.14, 8.0.6
    • Component/s: None
    • Labels:
      None

      Description

      {toc}

      xbcloud support for S3-compatible APIs

      Abstract

      We have had an experimental feature built into Percona XtraBackup for some time called xbcloud. This feature was originally intended to support OpenStack based clouds using Swift object storage. Since this feature was originally implemented, OpenStack has largely gone out of vogue, and AWS has continued to ascend. At this point in time all extent object storage systems provide AWS S3 compatible APIs or there are third-party shims available for them. For this reason, we have decided that supporting AWS S3 compatible APIs is the logical next step.

      Additionally, we are now building thorough best-practices based automation for MySQL into our Kubernetes Operators and the need for external storage of backups is of paramount importance.

      High Level Overview

      There are three targets which we'd like to be verified/tested against using xbcloud for managing backup data. These targets are Minio, AWS S3 itself, and Google Cloud Storage.

      Google Cloud Storage provides its own APIs, but the XML API is compatible with AWS S3 and it is sufficient at this point for our relevant NRE customer to support this. More details are in the Google Cloud Storage docs.

      At it's basic, xbcloud has three operations, put, get, and delete, broadly mapped to the use of HTTP syntax within the OpenStack Swift API. These operations are used for creating and storing a backup, retrieving and restoring a backup, and deleting a backup respectively. They also clearly map to similar operations within the AWS S3 API.

      Whatever we create needs to support arbitrary compatible endpoints.

      Additional New Features

      • xbcloud needs to support getting configuration from a file. This could be via a new section in my.cnf like xtrabackup already uses, or it could be a separate config. This is needed so keys don't need to be specified on command line in say a cron job.

      Some Specifics

      • xbcloud implements an option flag --storage which can be utilized here. We should add two additional options, one which says google and one which says s3. google can start off being mapped to the same code path, but in the future we may support the JSON / Full Google Cloud Storage API.
      • xbcloud expects a container/bucket to be precreated and specified, this is acceptable but similarly we need matching option language. So -s3-bucket= and google-bucket=. Same for [service]-endpoint, [service]-secret-key, and -[service]-access-key.
      • As already is the case, xbcloud should accept input via a pipe from xbstream so that it can be invoked as a pipeline with xtrabackup to stream directly to the cloud without needing local storage.

      Examples / Acceptance Criteria

      AWS S3 - Create Full Backup
      xtrabackup --backup --stream=xbstream --extra-lsndir=/tmp --target-dir=/tmp | \
      xbcloud put --storage=s3 \
          --s3-endpoint='s3.amazonaws.com' \
          --s3-access-key='YOUR-ACCESSKEYID' \
          --s3-secret-key='YOUR-SECRETACCESSKEY' \
          --s3-bucket='mysql_backups'
          --parallel=10 \
          ${date}-full_backup
      
      Minio - Create Full Backup
      xtrabackup --backup --stream=xbstream --extra-lsndir=/tmp --target-dir=/tmp | \
      xbcloud put --storage=s3 \
          --s3-endpoint='play.minio.io:9000' \
          --s3-access-key='YOUR-ACCESSKEYID' \
          --s3-secret-key='YOUR-SECRETACCESSKEY' \
          --s3-bucket='mysql_backups'
          --parallel=10 \
          ${date}-full_backup
      
      Google Cloud Storage - Create Full Backup
      xtrabackup --backup --stream=xbstream --extra-lsndir=/tmp --target-dir=/tmp | \
      xbcloud put --storage=google \
          --google-endpoint=`storage.googleapis.com` \
          --google-access-key='YOUR-ACCESSKEYID' \
          --google-secret-key='YOUR-SECRETACCESSKEY' \
          --google-bucket='mysql_backups'
          --parallel=10 \
          ${date}-full_backup
      

        Smart Checklist

          Attachments

            Issue Links

              Activity

                People

                • Assignee:
                  sergei.glushchenko Sergei Glushchenko
                  Reporter:
                  sergei.glushchenko Sergei Glushchenko
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  1 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Time Tracking

                    Estimated:
                    Original Estimate - Not Specified
                    Not Specified
                    Remaining:
                    Remaining Estimate - Not Specified
                    Not Specified
                    Logged:
                    Time Spent - 4 weeks, 1 day, 4 hours, 24 minutes
                    4w 1d 4h 24m