Details
-
Bug
-
Status: Done
-
Medium
-
Resolution: Fixed
-
1.11.0, 1.12.0
-
None
-
None
-
Yes
-
Yes
-
Yes
Description
In gke 1.19/1.20/1.21 operator 1.12 meet the error during downscale sharding to replica set
STR:
1. Apply default cr.yaml from release branch
2. Patch the psmdb
kubectl patch psmdb my-cluster-name --type=merge --patch '{"spec":{"sharding":
{"enabled": false}}}'
3. Operator logs:
{"level":"error","ts":1651575225.746852,"logger":"controller.psmdb-controller","msg":"Reconciler error","name":"my-cluster-name","namespace":"upgrade-cm-sharded-1","error":"reconcile StatefulSet for rs0: failed to run smartUpdate: failed to check active jobs: getting pbm object: create PBM connection to my-cluster-name-rs0-2.my-cluster-name-rs0.upgrade-cm-sharded-1.svc.cluster.local:27017,my-cluster-name-rs0-0.my-cluster-name-rs0.upgrade-cm-sharded-1.svc.cluster.local:27017,my-cluster-name-rs0-1.my-cluster-name-rs0.upgrade-cm-sharded-1.svc.cluster.local:27017: create mongo connection to configsvr with connection string 'mongodb://backup:Fwz7y6hJ90stMnM0@my-cluster-name-cfg-0.my-cluster-name-cfg.upgrade-cm-sharded-1.svc.cluster.local:27017,my-cluster-name-cfg-1.my-cluster-name-cfg.upgrade-cm-sharded-1.svc.cluster.local:27017,my-cluster-name-cfg-2.my-cluster-name-cfg.upgrade-cm-sharded-1.svc.cluster.local:27017/': mongo ping: server selection error: server selection timeout, current topology: { Type: Unknown, Servers: [{ Addr: my-cluster-name-cfg-0.my-cluster-name-cfg.upgrade-cm-sharded-1.svc.cluster.local:27017, Type: Unknown, Last error: connection() error occured during connection handshake: dial tcp: lookup my-cluster-name-cfg-0.my-cluster-name-cfg.upgrade-cm-sharded-1.svc.cluster.local on 10.47.240.10:53: no such host }, { Addr: my-cluster-name-cfg-1.my-cluster-name-cfg.upgrade-cm-sharded-1.svc.cluster.local:27017, Type: Unknown, Last error: connection() error occured during connection handshake: dial tcp: lookup my-cluster-name-cfg-1.my-cluster-name-cfg.upgrade-cm-sharded-1.svc.cluster.local on 10.47.240.10:53: no such host }, { Addr: my-cluster-name-cfg-2.my-cluster-name-cfg.upgrade-cm-sharded-1.svc.cluster.local:27017, Type: Unknown, Last error: connection() error occured during connection handshake: dial tcp: lookup my-cluster-name-cfg-2.my-cluster-name-cfg.upgrade-cm-sharded-1.svc.cluster.local on 10.47.240.10:53: no such host }, ] }","errorVerbose":"reconcile StatefulSet for rs0: failed to run smartUpdate: failed to check active jobs: getting pbm object: create PBM connection to my-cluster-name-rs0-2.my-cluster-name-rs0.upgrade-cm-sharded-1.svc.cluster.local:27017,my-cluster-name-rs0-0.my-cluster-name-rs0.upgrade-cm-sharded-1.svc.cluster.local:27017,my-cluster-name-rs0-1.my-cluster-name-rs0.upgrade-cm-sharded-1.svc.cluster.local:27017: create mongo connection to configsvr with connection string 'mongodb://backup:Fwz7y6hJ90stMnM0@my-cluster-name-cfg-0.my-cluster-name-cfg.upgrade-cm-sharded-1.svc.cluster.local:27017,my-cluster-name-cfg-1.my-cluster-name-cfg.upgrade-cm-sharded-1.svc.cluster.local:27017,my-cluster-name-cfg-2.my-cluster-name-cfg.upgrade-cm-sharded-1.svc.cluster.local:27017/': mongo ping: server selection error: server selection timeout, current topology: { Type: Unknown, Servers: [{ Addr: my-cluster-name-cfg-0.my-cluster-name-cfg.upgrade-cm-sharded-1.svc.cluster.local:27017, Type: Unknown, Last error: connection() error occured during connection handshake: dial tcp: lookup my-cluster-name-cfg-0.my-cluster-name-cfg.upgrade-cm-sharded-1.svc.cluster.local on 10.47.240.10:53: no such host }, { Addr: my-cluster-name-cfg-1.my-cluster-name-cfg.upgrade-cm-sharded-1.svc.cluster.local:27017, Type: Unknown, Last error: connection() error occured during connection handshake: dial tcp: lookup my-cluster-name-cfg-1.my-cluster-name-cfg.upgrade-cm-sharded-1.svc.cluster.local on 10.47.240.10:53: no such host }, { Addr: my-cluster-name-cfg-2.my-cluster-name-cfg.upgrade-cm-sharded-1.svc.cluster.local:27017, Type: Unknown, Last error: connection() error occured during connection handshake: dial tcp: lookup my-cluster-name-cfg-2.my-cluster-name-cfg.upgrade-cm-sharded-1.svc.cluster.local on 10.47.240.10:53: no such host }, ] }\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:409\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"}