Details
-
Bug
-
Status: Incomplete
-
Medium
-
Resolution: Unresolved
-
1.10.0
-
None
-
None
-
None
-
Yes
Description
Operator crashes right after startup, while it tries to create a new backup.
There are several psmdb-backups like:
Name: cron-mongodb-base-20211027220005-sd6sp Namespace: mynsp Labels: ancestor=mongodb-base-backup-daily-backup cluster=mongodb-base type=cron Annotations: <none> API Version: psmdb.percona.com/v1 Kind: PerconaServerMongoDBBackup Metadata: Creation Timestamp: 2021-10-27T22:00:05Z Finalizers: delete-backup Generate Name: cron-mongodb-base-20211027220005- Generation: 1 Managed Fields: API Version: psmdb.percona.com/v1 Manager: curl Operation: Update Time: 2021-10-27T22:00:05Z Resource Version: 21496698 UID: 93303349-0f52-4af4-a1c3-bf910d670e43 Spec: Psmdb Cluster: mongodb-base Storage Name: stor Events: <none>
Also several psmdb-backup resource have different errors:
oplog: read data: oplog has insufficient range, some records since the last saved ts
are missing. Run `pbm backup` to create a valid starting point for the PITR
and
failed to run backup: allowUnsafeConfigurations must be true to run backup on cluster with status initializing
Stack trace:
{"level":"info","ts":1634886699.5976052,"logger":"cmd","msg":"Git commit: e85383b8897afa6ae8b5b3c24d34667911955c51 Git branch: release-1-10-0"} {"level":"info","ts":1634886699.5976582,"logger":"cmd","msg":"Go Version: go1.16.8"} {"level":"info","ts":1634886699.5976696,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"} {"level":"info","ts":1634886699.59768,"logger":"cmd","msg":"operator-sdk Version: v0.17.2"} {"level":"info","ts":1634886699.597993,"logger":"leader","msg":"Trying to become the leader."} {"level":"info","ts":1634886702.0204363,"logger":"leader","msg":"Found existing lock with my name. I was likely restarted."} {"level":"info","ts":1634886702.0204904,"logger":"leader","msg":"Continuing as the leader."} {"level":"info","ts":1634886704.4251788,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":":8080"} {"level":"info","ts":1634886704.4254103,"logger":"cmd","msg":"Registering Components."} {"level":"info","ts":1634886704.430985,"logger":"controller_psmdb","msg":"server version","platform":"kubernetes","version":"v1.21.2"} {"level":"info","ts":1634886704.431491,"logger":"cmd","msg":"Starting the Cmd."} {"level":"info","ts":1634886704.4317548,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"} {"level":"info","ts":1634886704.4318128,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"psmdb-controller","source":"kind source: /, Kind="} {"level":"info","ts":1634886704.4319444,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"perconaservermongodbrestore-controller","source":"kind source: /, Kind="} {"level":"info","ts":1634886704.4321485,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"perconaservermongodbbackup-controller","source":"kind source: /, Kind="} {"level":"info","ts":1634886704.5323043,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"psmdb-controller"} {"level":"info","ts":1634886704.5324,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"psmdb-controller","worker count":1} {"level":"info","ts":1634886704.5329034,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"perconaservermongodbbackup-controller","source":"kind source: /, Kind="} {"level":"info","ts":1634886704.5329034,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"perconaservermongodbrestore-controller","source":"kind source: /, Kind="} {"level":"info","ts":1634886704.633241,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"perconaservermongodbrestore-controller"} {"level":"info","ts":1634886704.6333296,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"perconaservermongodbrestore-controller","worker count":1} {"level":"info","ts":1634886704.6334612,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"perconaservermongodbbackup-controller"} {"level":"info","ts":1634886704.7344542,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"perconaservermongodbbackup-controller","worker count":1} E1022 07:11:44.735477 1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference) goroutine 559 [running]: k8s.io/apimachinery/pkg/util/runtime.logPanic(0x1b37200, 0x2d5c760) /go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0xa6 k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0) /go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x86 panic(0x1b37200, 0x2d5c760) /usr/local/go/src/runtime/panic.go:965 +0x1b9 github.com/percona/percona-server-mongodb-operator/pkg/psmdb.GetAddr(0xc0006ae800, 0xc000747e00, 0x17, 0xc000a8a558, 0x8, 0xc000b00f00, 0xc000c02680) /go/src/github.com/percona/percona-server-mongodb-operator/pkg/psmdb/service.go:230 +0x178 github.com/percona/percona-server-mongodb-operator/pkg/psmdb.MongoHost(0x2158568, 0xc0006726f0, 0xc0006ae800, 0xc000a8a558, 0x8, 0x400, 0x19b31a3, 0x3, 0x1d720bf, 0x2, ...) /go/src/github.com/percona/percona-server-mongodb-operator/pkg/psmdb/service.go:210 +0x13a github.com/percona/percona-server-mongodb-operator/pkg/psmdb.GetReplsetAddrs(0x2158568, 0xc0006726f0, 0xc0006ae800, 0xc000a8a558, 0x8, 0x0, 0xc000157c00, 0x3, 0x3, 0x0, ...) /go/src/github.com/percona/percona-server-mongodb-operator/pkg/psmdb/service.go:194 +0x158 github.com/percona/percona-server-mongodb-operator/pkg/psmdb/backup.NewPBM(0x2158568, 0xc0006726f0, 0xc0006ae800, 0xc000791690, 0xb, 0xc000791730) /go/src/github.com/percona/percona-server-mongodb-operator/pkg/psmdb/backup/pbm.go:57 +0x246 github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).newBackup(0xc0005d2b10, 0xc000ce3340, 0x0, 0xc000791690, 0xb) /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/backup.go:27 +0xfa github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).Reconcile(0xc0005d2b10, 0xc000791690, 0xb, 0xc0005c33b0, 0x26, 0x134cdf700, 0x0, 0x0, 0x0) /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/perconaservermongodbbackup_controller.go:132 +0x35d sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000a68180, 0x1ba75c0, 0xc00053c5e0, 0xc000061700) /go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:256 +0x166 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000a68180, 0xc000aee800) /go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232 +0xb0 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker(...) /go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211 k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc0007c0010) /go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152 +0x5f k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0007c0010, 0x3b9aca00, 0x0, 0x1e47201, 0xc0005d5380) /go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153 +0x105 k8s.io/apimachinery/pkg/util/wait.Until(0xc0007c0010, 0x3b9aca00, 0xc0005d5380) /go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1 /go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:193 +0x32d panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference