Uploaded image for project: 'Percona Operator for MongoDB'
  1. Percona Operator for MongoDB
  2. K8SPSMDB-583

invalid memory address or nil pointer dereference at .newBackup GetAddr

Details

    • Bug
    • Status: Incomplete
    • Medium
    • Resolution: Unresolved
    • 1.10.0
    • None
    • None
    • None
    • Yes

    Description

      Operator crashes right after startup, while it tries to create a new backup.

      There are several psmdb-backups like:

      Name:         cron-mongodb-base-20211027220005-sd6sp
      Namespace:    mynsp
      Labels:       ancestor=mongodb-base-backup-daily-backup
                    cluster=mongodb-base
                    type=cron
      Annotations:  <none>
      API Version:  psmdb.percona.com/v1
      Kind:         PerconaServerMongoDBBackup
      Metadata:
        Creation Timestamp:  2021-10-27T22:00:05Z
        Finalizers:
          delete-backup
        Generate Name:  cron-mongodb-base-20211027220005-
        Generation:     1
        Managed Fields:
          API Version:  psmdb.percona.com/v1
          Manager:         curl
          Operation:       Update
          Time:            2021-10-27T22:00:05Z
        Resource Version:  21496698
        UID:               93303349-0f52-4af4-a1c3-bf910d670e43
      Spec:
        Psmdb Cluster:  mongodb-base
        Storage Name:   stor
      Events:           <none>
      

      Also several psmdb-backup resource have different errors:
      oplog: read data: oplog has insufficient range, some records since the last saved ts

      {1634767237 1}

      are missing. Run `pbm backup` to create a valid starting point for the PITR
      and
      failed to run backup: allowUnsafeConfigurations must be true to run backup on cluster with status initializing

      Stack trace:

      {"level":"info","ts":1634886699.5976052,"logger":"cmd","msg":"Git commit: e85383b8897afa6ae8b5b3c24d34667911955c51 Git branch: release-1-10-0"}
      {"level":"info","ts":1634886699.5976582,"logger":"cmd","msg":"Go Version: go1.16.8"}
      {"level":"info","ts":1634886699.5976696,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"}
      {"level":"info","ts":1634886699.59768,"logger":"cmd","msg":"operator-sdk Version: v0.17.2"}
      {"level":"info","ts":1634886699.597993,"logger":"leader","msg":"Trying to become the leader."}
      {"level":"info","ts":1634886702.0204363,"logger":"leader","msg":"Found existing lock with my name. I was likely restarted."}
      {"level":"info","ts":1634886702.0204904,"logger":"leader","msg":"Continuing as the leader."}
      {"level":"info","ts":1634886704.4251788,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":":8080"}
      {"level":"info","ts":1634886704.4254103,"logger":"cmd","msg":"Registering Components."}
      {"level":"info","ts":1634886704.430985,"logger":"controller_psmdb","msg":"server version","platform":"kubernetes","version":"v1.21.2"}
      {"level":"info","ts":1634886704.431491,"logger":"cmd","msg":"Starting the Cmd."}
      {"level":"info","ts":1634886704.4317548,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}
      {"level":"info","ts":1634886704.4318128,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"psmdb-controller","source":"kind source: /, Kind="}
      {"level":"info","ts":1634886704.4319444,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"perconaservermongodbrestore-controller","source":"kind source: /, Kind="}
      {"level":"info","ts":1634886704.4321485,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"perconaservermongodbbackup-controller","source":"kind source: /, Kind="}
      {"level":"info","ts":1634886704.5323043,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"psmdb-controller"}
      {"level":"info","ts":1634886704.5324,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"psmdb-controller","worker count":1}
      {"level":"info","ts":1634886704.5329034,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"perconaservermongodbbackup-controller","source":"kind source: /, Kind="}
      {"level":"info","ts":1634886704.5329034,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"perconaservermongodbrestore-controller","source":"kind source: /, Kind="}
      {"level":"info","ts":1634886704.633241,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"perconaservermongodbrestore-controller"}
      {"level":"info","ts":1634886704.6333296,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"perconaservermongodbrestore-controller","worker count":1}
      {"level":"info","ts":1634886704.6334612,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"perconaservermongodbbackup-controller"}
      {"level":"info","ts":1634886704.7344542,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"perconaservermongodbbackup-controller","worker count":1}
      E1022 07:11:44.735477       1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
      goroutine 559 [running]:
      k8s.io/apimachinery/pkg/util/runtime.logPanic(0x1b37200, 0x2d5c760)
              /go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0xa6
      k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
              /go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x86
      panic(0x1b37200, 0x2d5c760)
              /usr/local/go/src/runtime/panic.go:965 +0x1b9
      github.com/percona/percona-server-mongodb-operator/pkg/psmdb.GetAddr(0xc0006ae800, 0xc000747e00, 0x17, 0xc000a8a558, 0x8, 0xc000b00f00, 0xc000c02680)
              /go/src/github.com/percona/percona-server-mongodb-operator/pkg/psmdb/service.go:230 +0x178
      github.com/percona/percona-server-mongodb-operator/pkg/psmdb.MongoHost(0x2158568, 0xc0006726f0, 0xc0006ae800, 0xc000a8a558, 0x8, 0x400, 0x19b31a3, 0x3, 0x1d720bf, 0x2, ...)
              /go/src/github.com/percona/percona-server-mongodb-operator/pkg/psmdb/service.go:210 +0x13a
      github.com/percona/percona-server-mongodb-operator/pkg/psmdb.GetReplsetAddrs(0x2158568, 0xc0006726f0, 0xc0006ae800, 0xc000a8a558, 0x8, 0x0, 0xc000157c00, 0x3, 0x3, 0x0, ...)
              /go/src/github.com/percona/percona-server-mongodb-operator/pkg/psmdb/service.go:194 +0x158
      github.com/percona/percona-server-mongodb-operator/pkg/psmdb/backup.NewPBM(0x2158568, 0xc0006726f0, 0xc0006ae800, 0xc000791690, 0xb, 0xc000791730)
              /go/src/github.com/percona/percona-server-mongodb-operator/pkg/psmdb/backup/pbm.go:57 +0x246
      github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).newBackup(0xc0005d2b10, 0xc000ce3340, 0x0, 0xc000791690, 0xb)
              /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/backup.go:27 +0xfa
      github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup.(*ReconcilePerconaServerMongoDBBackup).Reconcile(0xc0005d2b10, 0xc000791690, 0xb, 0xc0005c33b0, 0x26, 0x134cdf700, 0x0, 0x0, 0x0)        /go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodbbackup/perconaservermongodbbackup_controller.go:132 +0x35d
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc000a68180, 0x1ba75c0, 0xc00053c5e0, 0xc000061700)
              /go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:256 +0x166
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc000a68180, 0xc000aee800)
              /go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:232 +0xb0
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker(...)
              /go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:211
      k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc0007c0010)
              /go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152 +0x5f
      k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0007c0010, 0x3b9aca00, 0x0, 0x1e47201, 0xc0005d5380)
              /go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153 +0x105
      k8s.io/apimachinery/pkg/util/wait.Until(0xc0007c0010, 0x3b9aca00, 0xc0005d5380)
              /go/src/github.com/percona/percona-server-mongodb-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d
      created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
              /go/src/github.com/percona/percona-server-mongodb-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:193 +0x32d
      panic: runtime error: invalid memory address or nil pointer dereference [recovered]
              panic: runtime error: invalid memory address or nil pointer dereference
      
      

      Attachments

        Activity

          People

            andrii.dema Andrii Dema
            nickolay.ihalainen Nickolay Ihalainen
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:

              Smart Checklist