Uploaded image for project: 'Percona Operator for MongoDB'
  1. Percona Operator for MongoDB
  2. K8SPSMDB-660

backup error - starting deadline exceeded

Details

    • Bug
    • Status: Done
    • Medium
    • Resolution: Fixed
    • 1.11.0
    • 1.13.0
    • None
    • Yes
    • Yes

    Description

      When starting a backup it can happen that the deadline to start a backup can be exceeded especially in the sharding environment because of many nodes.
      We have set some deadline of 40sec and if the backup is not started by then it will be marked as failed (although it can actually succeed).
      This deadline is set here: https://github.com/percona/percona-server-mongodb-operator/blob/main/pkg/controller/perconaservermongodbbackup/backup.go#L15

      The error can look like this:

      apiVersion: psmdb.percona.com/v1
      kind: PerconaServerMongoDBBackup
      metadata:
        annotations:
          kubectl.kubernetes.io/last-applied-configuration: |
            {"apiVersion":"psmdb.percona.com/v1","kind":"PerconaServerMongoDBBackup","metadata":{"annotations":{},"name":"backup1","namespace":"mongo"},"spec":{"psmdbCluster":"my-cluster-name","storageName":"s3-eu-west-2"}}
        creationTimestamp: "2022-03-21T09:23:12Z"
        generation: 1
        name: backup1
        namespace: mongo
        resourceVersion: "2189522"
        uid: 89007408-3439-4b52-a406-af71643f7b40
      spec:
        psmdbCluster: my-cluster-name
        storageName: s3-eu-west-2
      status:
        azure:
          credentialsSecret: ""
        destination: "2022-03-21T09:23:33Z"
        error: starting deadline exceeded
        lastTransition: "2022-03-21T09:23:34Z"
        pbmName: "2022-03-21T09:23:33Z"
        s3:
          bucket: <bucket>
          credentialsSecret: my-cluster-name-backup-s3
          region: eu-west-2
        start: "2022-03-21T09:23:34Z"
        state: error
        storageName: s3-eu-west-2
      

      Attachments

        Issue Links

          Activity

            People

              ege.gunes Ege Gunes
              tomislav.plavcic@percona.com Tomislav Plavcic
              Votes:
              2 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Smart Checklist