Uploaded image for project: 'Percona Operator for MongoDB'
  1. Percona Operator for MongoDB
  2. K8SPSMDB-660

backup error - starting deadline exceeded

Details

    • Bug
    • Status: Done
    • Medium
    • Resolution: Fixed
    • 1.11.0
    • 1.13.0
    • None
    • Yes
    • Yes

    Description

      When starting a backup it can happen that the deadline to start a backup can be exceeded especially in the sharding environment because of many nodes.
      We have set some deadline of 40sec and if the backup is not started by then it will be marked as failed (although it can actually succeed).
      This deadline is set here: https://github.com/percona/percona-server-mongodb-operator/blob/main/pkg/controller/perconaservermongodbbackup/backup.go#L15

      The error can look like this:

      apiVersion: psmdb.percona.com/v1
      kind: PerconaServerMongoDBBackup
      metadata:
        annotations:
          kubectl.kubernetes.io/last-applied-configuration: |
            {"apiVersion":"psmdb.percona.com/v1","kind":"PerconaServerMongoDBBackup","metadata":{"annotations":{},"name":"backup1","namespace":"mongo"},"spec":{"psmdbCluster":"my-cluster-name","storageName":"s3-eu-west-2"}}
        creationTimestamp: "2022-03-21T09:23:12Z"
        generation: 1
        name: backup1
        namespace: mongo
        resourceVersion: "2189522"
        uid: 89007408-3439-4b52-a406-af71643f7b40
      spec:
        psmdbCluster: my-cluster-name
        storageName: s3-eu-west-2
      status:
        azure:
          credentialsSecret: ""
        destination: "2022-03-21T09:23:33Z"
        error: starting deadline exceeded
        lastTransition: "2022-03-21T09:23:34Z"
        pbmName: "2022-03-21T09:23:33Z"
        s3:
          bucket: <bucket>
          credentialsSecret: my-cluster-name-backup-s3
          region: eu-west-2
        start: "2022-03-21T09:23:34Z"
        state: error
        storageName: s3-eu-west-2
      

      Attachments

        Activity

          People

            ege.gunes Ege Gunes
            tomislav.plavcic@percona.com Tomislav Plavcic
            Votes:
            2 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Smart Checklist