Uploaded image for project: 'Percona Operator for MongoDB'
  1. Percona Operator for MongoDB
  2. K8SPSMDB-568

upgrading to 5.0 fails when using upgradeOptions:apply

Details

    • Bug
    • Status: Done
    • Medium
    • Resolution: Fixed
    • 1.10.0
    • 1.11.0
    • None
    • Yes
    • Yes
    • Yes

    Description

      I was trying to do major upgrade from 4.0 to 5.0 using "upgradeOptions:apply" set to something like "4.0-recommended" or "4.0-latest" and it upgrades to 4.4 (from 4.2) setting the FCV to 4.4 also, but it doesn't want to upgrade to 5.0.

      When it came to 5.0 upgrade it is throwing this error:

      {"level":"error","ts":1632407880.0319288,"logger":"controller_psmdb","msg":"failed to ensure version","error":"failed to check if major update requested: can't upgrade to 5.0 with FCV set to 4.4","errorVerbose":"can't upgrade to 5.0 with FCV set to 4.4\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.majorUpgradeRequested\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/version.go:182\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).ensureVersion\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/version.go:221\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).sheduleEnsureVersion.func1\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/version.go:79\ngithub.com/robfig/cron/v3.FuncJob.Run\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/github.com/robfig/cron/v3/cron.go:136\ngithub.com/robfig/cron/v3.(*Cron).startJob.func1\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/github.com/robfig/cron/v3/cron.go:312\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371\nfailed to check if major update requested\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).ensureVersion\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/version.go:223\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).sheduleEnsureVersion.func1\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/version.go:79\ngithub.com/robfig/cron/v3.FuncJob.Run\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/github.com/robfig/cron/v3/cron.go:136\ngithub.com/robfig/cron/v3.(*Cron).startJob.func1\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/github.com/robfig/cron/v3/cron.go:312\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).sheduleEnsureVersion.func1\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/version.go:81\ngithub.com/robfig/cron/v3.FuncJob.Run\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/github.com/robfig/cron/v3/cron.go:136\ngithub.com/robfig/cron/v3.(*Cron).startJob.func1\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/github.com/robfig/cron/v3/cron.go:312"}
      

      Steps to reproduce:
      1. start with something like:

        updateStrategy: SmartUpdate
        upgradeOptions:
          apply: 4.4-latest
          schedule: '* * * * *'
          setFCV: true
          versionServiceEndpoint: https://check-dev.percona.com
      

      2. patch:

      kubectl patch psmdb my-cluster-name --type=merge --patch '{"spec": {"upgradeOptions":{ "apply": "5.0-latest" }}}'
      

      I have managed to upgrade by patching the 5.0 image directly, but FCV version was left on 4.4 even if setFCV option was set to "true" (to be honest I'm not sure that FCV should be changed if we patch the image directly, but it needs to be investigated):

      + kubectl run -i --rm --tty percona-client-24290 --image=percona/percona-server-mongodb:4.4 --restart=Never -- mongo 'mongodb://backup:backup12[email protected]/admin?ssl=false&authSource=admin&replicaSet=rs0'
      If you don't see a command prompt, try pressing enter.
      
      rs0:PRIMARY> db.adminCommand({ getParameter: 1, featureCompatibilityVersion: 1 })
      {
              "featureCompatibilityVersion" : {
                      "version" : "4.4"
              },
              "ok" : 1,
              "$gleStats" : {
                      "lastOpTime" : Timestamp(0, 0),
                      "electionId" : ObjectId("7fffffff0000000000000004")
              },
              "lastCommittedOpTime" : Timestamp(1632408590, 2),
              "$configServerState" : {
                      "opTime" : {
                              "ts" : Timestamp(1632408596, 7),
                              "t" : NumberLong(-1)
                      }
              },
              "$clusterTime" : {
                      "clusterTime" : Timestamp(1632408596, 7),
                      "signature" : {
                              "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
                              "keyId" : NumberLong(0)
                      }
              },
              "operationTime" : Timestamp(1632408590, 2)
      }
      

      But I have managed to change the FCV to 5.0 manually.

      Attachments

        Issue Links

          Activity

            People

              mykola.marzhan Mykola Marzhan (Inactive)
              tomislav.plavcic@percona.com Tomislav Plavcic
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Smart Checklist