Details
-
Bug
-
Status: Done
-
Medium
-
Resolution: Fixed
-
None
-
None
-
Yes
Description
If I delete the CR, the pods start to terminate, but then they start to re-init again (the same pods which were in termination).
This runs for some time in a loop until eventually it succeeds, but it doesn't look right and cluster deletion takes a lot of time.
To reproduce best use a sharding cluster with 2 or even 3 shards and with 3 pods per shard just so that it is easily visible.
For PSMDB image I was using: "percona/percona-server-mongodb:5.0.6-5" and for operator image I was using: "perconalab/percona-server-mongodb-operator:main"
Some possibly related messages:
{"level":"info","ts":1648024784.5433416,"logger":"controller_psmdb","msg":"Replset size will be changed from 1 to 3 due to safe config"}
{"level":"info","ts":1648024784.5433846,"logger":"controller_psmdb","msg":"Set allowUnsafeConfigurations=true to disable safe configuration"}
{"level":"info","ts":1648024784.5434053,"logger":"controller_psmdb","msg":"Replset size will be changed from 1 to 3 due to safe config"}
{"level":"info","ts":1648024784.5434093,"logger":"controller_psmdb","msg":"Set allowUnsafeConfigurations=true to disable safe configuration"}
{"level":"error","ts":1648024784.5436356,"logger":"controller_psmdb","msg":"failed to run finalizer","finalizer":"delete-psmdb-pods-in-order","error":"waiting pods to be deleted","errorVerbose":"waiting pods to be deleted\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).deletePSMDBPods\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/finalizers.go:65\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).checkFinalizers\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/finalizers.go:23\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:228\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227\nruntime.goexit\n\t/
usr/local/go/src/runtime/asm_amd64.s:1581","stacktrace":"github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).checkFinalizers\n\t/go/src/github.c
om/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/finalizers.go:29\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMong
oDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:228\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Recon
cile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Control
ler).reconcileHandler\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/con
troller.(*Controller).processNextWorkItem\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runt
ime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":1648024784.8784163,"logger":"controller_psmdb","msg":"failed to run finalizer","finalizer":"delete-psmdb-pods-in-order","error":"waiting statefulsets to be resized","errorVerbose":"waiting s
tatefulsets to be resized\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).deletePSMDBPods\n\t/go/src/github.com/percona/percona-server-mong
odb-operator/pkg/controller/perconaservermongodb/finalizers.go:73\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).checkFinalizers\n\t/go/sr
c/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/finalizers.go:23\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePercona
ServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:228\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Control
ler).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller
.(*Controller).reconcileHandler\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/in
ternal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io/contr
oller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227\nrun
time.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581","stacktrace":"github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).checkFinalizers\n\t
/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/finalizers.go:29\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcileP
erconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:228\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*C
ontroller).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/cont
roller.(*Controller).reconcileHandler\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/
pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266\nsigs.k8s.io
/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:22
7"}
Notice the resizing message and error about delete-psmdb-pods-in-order finalizer.
Here's the pod list output:
NAME READY STATUS RESTARTS AGE my-cluster-name-cfg-0 2/2 Running 0 13m my-cluster-name-cfg-1 2/2 Running 0 12m my-cluster-name-cfg-2 2/2 Running 0 12m my-cluster-name-mongos-0 1/1 Running 0 11m my-cluster-name-mongos-1 1/1 Running 0 11m my-cluster-name-mongos-2 1/1 Running 0 11m my-cluster-name-rs0-0 2/2 Running 0 13m my-cluster-name-rs0-1 0/2 Terminating 0 4s my-cluster-name-rs1-0 2/2 Running 0 13m my-cluster-name-rs1-1 0/2 Terminating 0 6s percona-server-mongodb-operator-686b7f6cb6-lr99d 1/1 Running 0 13m NAME READY STATUS RESTARTS AGE my-cluster-name-cfg-0 2/2 Running 0 13m my-cluster-name-cfg-1 2/2 Running 0 12m my-cluster-name-cfg-2 2/2 Running 0 12m my-cluster-name-mongos-0 1/1 Running 0 11m my-cluster-name-mongos-1 1/1 Running 0 11m my-cluster-name-mongos-2 1/1 Running 0 11m my-cluster-name-rs0-0 2/2 Running 0 13m my-cluster-name-rs0-1 0/2 Terminating 0 9s my-cluster-name-rs1-0 2/2 Running 0 13m my-cluster-name-rs1-1 0/2 Init:0/1 0 1s percona-server-mongodb-operator-686b7f6cb6-lr99d 1/1 Running 0 14m NAME READY STATUS RESTARTS AGE my-cluster-name-cfg-0 2/2 Running 0 13m my-cluster-name-cfg-1 2/2 Running 0 13m my-cluster-name-cfg-2 2/2 Running 0 12m my-cluster-name-mongos-0 1/1 Running 0 11m my-cluster-name-mongos-1 1/1 Running 0 11m my-cluster-name-mongos-2 1/1 Running 0 11m my-cluster-name-rs0-0 2/2 Running 0 13m my-cluster-name-rs0-1 0/2 Terminating 0 4s my-cluster-name-rs1-0 2/2 Running 0 13m my-cluster-name-rs1-1 0/2 Terminating 0 6s percona-server-mongodb-operator-686b7f6cb6-lr99d 1/1 Running 0 14m NAME READY STATUS RESTARTS AGE my-cluster-name-cfg-0 2/2 Running 0 14m my-cluster-name-cfg-1 2/2 Running 0 13m my-cluster-name-cfg-2 2/2 Running 0 12m my-cluster-name-mongos-0 1/1 Running 0 12m my-cluster-name-mongos-1 1/1 Running 0 12m my-cluster-name-mongos-2 1/1 Running 0 11m my-cluster-name-rs0-0 2/2 Running 0 14m my-cluster-name-rs0-1 0/2 Terminating 0 8s my-cluster-name-rs1-0 2/2 Running 0 14m my-cluster-name-rs1-1 0/2 Pending 0 0s percona-server-mongodb-operator-686b7f6cb6-lr99d 1/1 Running 0 14m NAME READY STATUS RESTARTS AGE my-cluster-name-cfg-0 2/2 Running 0 14m my-cluster-name-cfg-1 2/2 Running 0 13m my-cluster-name-cfg-2 2/2 Running 0 12m my-cluster-name-mongos-0 1/1 Running 0 12m my-cluster-name-mongos-1 1/1 Running 0 12m my-cluster-name-mongos-2 1/1 Running 0 11m my-cluster-name-rs0-0 2/2 Running 0 14m my-cluster-name-rs0-1 0/2 Terminating 0 3s my-cluster-name-rs1-0 2/2 Running 0 14m my-cluster-name-rs1-1 0/2 Terminating 0 5s percona-server-mongodb-operator-686b7f6cb6-lr99d 1/1 Running 0 14m
Attachments
Issue Links
- relates to
-
-
- Done
-