Uploaded image for project: 'Percona Operator for MongoDB'
  1. Percona Operator for MongoDB
  2. K8SPSMDB-686

Failed to downscale sharding to replica set

Details

    • Bug
    • Status: Done
    • Medium
    • Resolution: Fixed
    • 1.11.0, 1.12.0
    • 1.13.0
    • None
    • None
    • Yes
    • Yes
    • Yes

    Description

      In gke 1.19/1.20/1.21 operator 1.12 meet the error during downscale sharding to replica set 
      STR: 
      1. Apply default cr.yaml from release branch
      2. Patch the psmdb 

      kubectl patch psmdb my-cluster-name --type=merge --patch '{"spec":{"sharding":

      {"enabled": false}

      }}'
      3. Operator logs: 

      {"level":"error","ts":1651575225.746852,"logger":"controller.psmdb-controller","msg":"Reconciler error","name":"my-cluster-name","namespace":"upgrade-cm-sharded-1","error":"reconcile StatefulSet for rs0: failed to run smartUpdate: failed to check active jobs: getting pbm object: create PBM connection to my-cluster-name-rs0-2.my-cluster-name-rs0.upgrade-cm-sharded-1.svc.cluster.local:27017,my-cluster-name-rs0-0.my-cluster-name-rs0.upgrade-cm-sharded-1.svc.cluster.local:27017,my-cluster-name-rs0-1.my-cluster-name-rs0.upgrade-cm-sharded-1.svc.cluster.local:27017: create mongo connection to configsvr with connection string 'mongodb://backup:[email protected]de-cm-sharded-1.svc.cluster.local:27017,my-cluster-name-cfg-1.my-cluster-name-cfg.upgrade-cm-sharded-1.svc.cluster.local:27017,my-cluster-name-cfg-2.my-cluster-name-cfg.upgrade-cm-sharded-1.svc.cluster.local:27017/': mongo ping: server selection error: server selection timeout, current topology: { Type: Unknown, Servers: [{ Addr: my-cluster-name-cfg-0.my-cluster-name-cfg.upgrade-cm-sharded-1.svc.cluster.local:27017, Type: Unknown, Last error: connection() error occured during connection handshake: dial tcp: lookup my-cluster-name-cfg-0.my-cluster-name-cfg.upgrade-cm-sharded-1.svc.cluster.local on 10.47.240.10:53: no such host }, { Addr: my-cluster-name-cfg-1.my-cluster-name-cfg.upgrade-cm-sharded-1.svc.cluster.local:27017, Type: Unknown, Last error: connection() error occured during connection handshake: dial tcp: lookup my-cluster-name-cfg-1.my-cluster-name-cfg.upgrade-cm-sharded-1.svc.cluster.local on 10.47.240.10:53: no such host }, { Addr: my-cluster-name-cfg-2.my-cluster-name-cfg.upgrade-cm-sharded-1.svc.cluster.local:27017, Type: Unknown, Last error: connection() error occured during connection handshake: dial tcp: lookup my-cluster-name-cfg-2.my-cluster-name-cfg.upgrade-cm-sharded-1.svc.cluster.local on 10.47.240.10:53: no such host }, ] }","errorVerbose":"reconcile StatefulSet for rs0: failed to run smartUpdate: failed to check active jobs: getting pbm object: create PBM connection to my-cluster-name-rs0-2.my-cluster-name-rs0.upgrade-cm-sharded-1.svc.cluster.local:27017,my-cluster-name-rs0-0.my-cluster-name-rs0.upgrade-cm-sharded-1.svc.cluster.local:27017,my-cluster-name-rs0-1.my-cluster-name-rs0.upgrade-cm-sharded-1.svc.cluster.local:27017: create mongo connection to configsvr with connection string 'mongodb://backup:[email protected]de-cm-sharded-1.svc.cluster.local:27017,my-cluster-name-cfg-1.my-cluster-name-cfg.upgrade-cm-sharded-1.svc.cluster.local:27017,my-cluster-name-cfg-2.my-cluster-name-cfg.upgrade-cm-sharded-1.svc.cluster.local:27017/': mongo ping: server selection error: server selection timeout, current topology: { Type: Unknown, Servers: [{ Addr: my-cluster-name-cfg-0.my-cluster-name-cfg.upgrade-cm-sharded-1.svc.cluster.local:27017, Type: Unknown, Last error: connection() error occured during connection handshake: dial tcp: lookup my-cluster-name-cfg-0.my-cluster-name-cfg.upgrade-cm-sharded-1.svc.cluster.local on 10.47.240.10:53: no such host }, { Addr: my-cluster-name-cfg-1.my-cluster-name-cfg.upgrade-cm-sharded-1.svc.cluster.local:27017, Type: Unknown, Last error: connection() error occured during connection handshake: dial tcp: lookup my-cluster-name-cfg-1.my-cluster-name-cfg.upgrade-cm-sharded-1.svc.cluster.local on 10.47.240.10:53: no such host }, { Addr: my-cluster-name-cfg-2.my-cluster-name-cfg.upgrade-cm-sharded-1.svc.cluster.local:27017, Type: Unknown, Last error: connection() error occured during connection handshake: dial tcp: lookup my-cluster-name-cfg-2.my-cluster-name-cfg.upgrade-cm-sharded-1.svc.cluster.local on 10.47.240.10:53: no such host }, ] }\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:409\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"}
      

      Attachments

        Activity

          People

            Unassigned Unassigned
            natalia.marukovich Natalia Marukovich
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Smart Checklist