Details
-
Bug
-
Status: Done
-
Critical
-
Resolution: Fixed
-
None
-
Yes
-
1
-
Yes
-
Yes
-
No
-
Server Features
-
Description
User impact:
Restore fails if a user has enabled PITR
Steps to reproduce:
- Start PMM Server Backup GA
- setup mongo replica set for backups
- setup s3 location and enable PITR every 2 mins
- wait for a snapshot and wait for the timeranges for restore
- trigger restore and explore pmm-agent logs
Actual result:
- restore failed with error in pmm-agent logs: Job terminated with error: another operation in progress
Expected result:
- Restore should be successful without any issues
Note:
pmm-agent log:
INFO[2022-10-25T10:55:48.584+00:00] Job started. component=runner id=/job_id/ce66905b-6cb4-4961-a205-fd8f5b7d6614 type=mongodb_restore INFO[2022-10-25T10:55:48.585+00:00] Configuring S3 location. id=/job_id/ce66905b-6cb4-4961-a205-fd8f5b7d6614 name=vasyl_pitr type=mongodb_restore INFO[2022-10-25T10:55:48.742+00:00] Waiting for no running pbm operations. id=/job_id/ce66905b-6cb4-4961-a205-fd8f5b7d6614 name=vasyl_pitr type=mongodb_restore INFO[2022-10-25T10:55:54.676+00:00] Finding backup entity name. id=/job_id/ce66905b-6cb4-4961-a205-fd8f5b7d6614 name=vasyl_pitr type=mongodb_restore INFO[2022-10-25T10:55:55.853+00:00] starting backup restore for: 2022-10-25T10:55:06Z. id=/job_id/ce66905b-6cb4-4961-a205-fd8f5b7d6614 name=vasyl_pitr type=mongodb_restore WARN[2022-10-25T10:55:55.968+00:00] Job terminated with error: another operation in progress github.com/percona/pmm/agent/runner/jobs.execPBMCommand /tmp/go/src/github.com/percona/pmm/agent/runner/jobs/pbm_helpers.go:179 github.com/percona/pmm/agent/runner/jobs.(*MongoDBRestoreJob).startRestore /tmp/go/src/github.com/percona/pmm/agent/runner/jobs/mongodb_restore_job.go:170 github.com/percona/pmm/agent/runner/jobs.(*MongoDBRestoreJob).Run /tmp/go/src/github.com/percona/pmm/agent/runner/jobs/mongodb_restore_job.go:109 github.com/percona/pmm/agent/runner.(*Runner).handleJob.func1 /tmp/go/src/github.com/percona/pmm/agent/runner/runner.go:183 runtime/pprof.Do /usr/local/go/src/runtime/pprof/runtime.go:40 runtime.goexit /usr/local/go/src/runtime/asm_amd64.s:1594 pbm restore error: another operation in progress github.com/percona/pmm/agent/runner/jobs.(*MongoDBRestoreJob).startRestore /tmp/go/src/github.com/percona/pmm/agent/runner/jobs/mongodb_restore_job.go:173 github.com/percona/pmm/agent/runner/jobs.(*MongoDBRestoreJob).Run /tmp/go/src/github.com/percona/pmm/agent/runner/jobs/mongodb_restore_job.go:109 github.com/percona/pmm/agent/runner.(*Runner).handleJob.func1 /tmp/go/src/github.com/percona/pmm/agent/runner/runner.go:183 runtime/pprof.Do /usr/local/go/src/runtime/pprof/runtime.go:40 runtime.goexit /usr/local/go/src/runtime/asm_amd64.s:1594 failed to start backup restore github.com/percona/pmm/agent/runner/jobs.(*MongoDBRestoreJob).Run /tmp/go/src/github.com/percona/pmm/agent/runner/jobs/mongodb_restore_job.go:111 github.com/percona/pmm/agent/runner.(*Runner).handleJob.func1 /tmp/go/src/github.com/percona/pmm/agent/runner/runner.go:183 runtime/pprof.Do /usr/local/go/src/runtime/pprof/runtime.go:40 runtime.goexit /usr/local/go/src/runtime/asm_amd64.s:1594 component=runner id=/job_id/ce66905b-6cb4-4961-a205-fd8f5b7d6614 type=mongodb_restore INFO[2022-10-25T10:55:55.969+00:00] Job finished. component=runner duration=7.384989921s id=/job_id/ce66905b-6cb4-4961-a205-fd8f5b7d6614 type=mongodb_restore