We have had an experimental feature built into Percona XtraBackup for some time called xbcloud. This feature was originally intended to support OpenStack based clouds using Swift object storage. Since this feature was originally implemented, OpenStack has largely gone out of vogue, and AWS has continued to ascend. At this point in time all extent object storage systems provide AWS S3 compatible APIs or there are third-party shims available for them. For this reason, we have decided that supporting AWS S3 compatible APIs is the logical next step.
Additionally, we are now building thorough best-practices based automation for MySQL into our Kubernetes Operators and the need for external storage of backups is of paramount importance.
Google Cloud Storage provides its own APIs, but the XML API is compatible with AWS S3 and it is sufficient at this point for our relevant NRE customer to support this. More details are in the Google Cloud Storage docs.
At it's basic, xbcloud has three operations, put, get, and delete, broadly mapped to the use of HTTP syntax within the OpenStack Swift API. These operations are used for creating and storing a backup, retrieving and restoring a backup, and deleting a backup respectively. They also clearly map to similar operations within the AWS S3 API.
Whatever we create needs to support arbitrary compatible endpoints.
- xbcloud needs to support getting configuration from a file. This could be via a new section in my.cnf like xtrabackup already uses, or it could be a separate config. This is needed so keys don't need to be specified on command line in say a cron job.
- xbcloud implements an option flag --storage which can be utilized here. We should add two additional options, one which says google and one which says s3. google can start off being mapped to the same code path, but in the future we may support the JSON / Full Google Cloud Storage API.
- xbcloud expects a container/bucket to be precreated and specified, this is acceptable but similarly we need matching option language. So -
s3-bucket=and google-bucket=. Same for [service]-endpoint, [service]-secret-key, and -[service]-access-key.
- As already is the case, xbcloud should accept input via a pipe from xbstream so that it can be invoked as a pipeline with xtrabackup to stream directly to the cloud without needing local storage.