Uploaded image for project: 'Percona Monitoring and Management'
  1. Percona Monitoring and Management
  2. PMM-8573

Inventory page returns Internal server error after upgrade

    XMLWordPrintable

Details

    • Bug
    • Status: Done
    • High
    • Resolution: Fixed
    • 2.21.0
    • 2.21.0
    • None
    • Yes
    • Yes
    • Yes
    • C/S Core

    Description

      User impact: User see the error message and can't see any agents

      STR:

      1. Install PMM 2.20.0
      2. Enable experimental repositories for pmm2-client and for PMM server packages
      3. Upgrade to 2.21.0 via UI
      4. Open PMM Inventory page after upgrade:

      Actual result: Internal error message appears and there is no any agents

      Expected result: Agents are displayed and no any errors

      in pmm-managed.log:

       
INFO[2021-08-09T14:41:37.192+00:00] RPC /server.Server/Readiness done in 3.188305ms.  request=e687c3e8-f91f-11eb-bbcc-0242ac110002
INFO[2021-08-09T14:41:37.544+00:00] Starting Stream /agent.Agent/Connect ...      agent_id=pmm-server request=e6be0ad2-f91f-11eb-bbcc-0242ac110002
WARN[2021-08-09T14:41:37.549+00:00] Failed to authenticate connected pmm-agent &{ID:pmm-server Version:2.20.0 MetricsPort:0}.  agent_id=pmm-server request=e6be0ad2-f91f-11eb-bbcc-0242ac110002
ERRO[2021-08-09T14:41:37.549+00:00] Stream /agent.Agent/Connect done in 4.497256ms with unexpected error: pq: column agents.agent_password does not exist
github.com/percona/pmm-managed/models.FindAgentByID
	/home/builder/rpm/BUILD/pmm-managed-f581ced32bf5231e0bc296d19e005e90ead70d1f/src/github.com/percona/pmm-managed/models/agent_helpers.go:210
github.com/percona/pmm-managed/services/agents.authenticate
	/home/builder/rpm/BUILD/pmm-managed-f581ced32bf5231e0bc296d19e005e90ead70d1f/src/github.com/percona/pmm-managed/services/agents/registry.go:213
github.com/percona/pmm-managed/services/agents.(*Registry).register.func1
	/home/builder/rpm/BUILD/pmm-managed-f581ced32bf5231e0bc296d19e005e90ead70d1f/src/github.com/percona/pmm-managed/services/agents/registry.go:163
gopkg.in/reform%2ev1.(*DB).InTransactionContext
	/home/builder/go/pkg/mod/gopkg.in/[email protected]/db.go:93
gopkg.in/reform%2ev1.(*DB).InTransaction
	/home/builder/go/pkg/mod/gopkg.in/[email protected]/db.go:74
github.com/percona/pmm-managed/services/agents.(*Registry).register
	/home/builder/rpm/BUILD/pmm-managed-f581ced32bf5231e0bc296d19e005e90ead70d1f/src/github.com/percona/pmm-managed/services/agents/registry.go:162
github.com/percona/pmm-managed/services/agents.(*Handler).Run
	/home/builder/rpm/BUILD/pmm-managed-f581ced32bf5231e0bc296d19e005e90ead70d1f/src/github.com/percona/pmm-managed/services/agents/handler.go:70
github.com/percona/pmm-managed/services/agents/grpc.(*agentServer).Connect
	/home/builder/rpm/BUILD/pmm-managed-f581ced32bf5231e0bc296d19e005e90ead70d1f/src/github.com/percona/pmm-managed/services/agents/grpc/agent_server.go:39
github.com/percona/pmm/api/agentpb._Agent_Connect_Handler
	/home/builder/go/pkg/mod/github.com/percona/[email protected]/api/agentpb/agent.pb.go:7212
github.com/grpc-ecosystem/go-grpc-middleware/validator.StreamServerInterceptor.func1
	/home/builder/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/validator/validator.go:41
github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1
	/home/builder/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49
github.com/percona/pmm-managed/utils/interceptors.StreamServiceEnabledInterceptor.func1
	/home/builder/rpm/BUILD/pmm-managed-f581ced32bf5231e0bc296d19e005e90ead70d1f/src/github.com/percona/pmm-managed/utils/interceptors/service.go:50
github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1
	/home/builder/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49
github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).StreamServerInterceptor.func1
	/home/builder/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/server_metrics.go:121
github.com/percona/pmm-managed/utils/interceptors.Stream.func1
	/home/builder/rpm/BUILD/pmm-managed-f581ced32bf5231e0bc296d19e005e90ead70d1f/src/github.com/percona/pmm-managed/utils/interceptors/interceptors.go:125
github.com/percona/pmm-managed/utils/interceptors.logRequest
	/home/builder/rpm/BUILD/pmm-managed-f581ced32bf5231e0bc296d19e005e90ead70d1f/src/github.com/percona/pmm-managed/utils/interceptors/interceptors.go:78
github.com/percona/pmm-managed/utils/interceptors.Stream
	/home/builder/rpm/BUILD/pmm-managed-f581ced32bf5231e0bc296d19e005e90ead70d1f/src/github.com/percona/pmm-managed/utils/interceptors/interceptors.go:122
github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1
	/home/builder/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49
github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1
	/home/builder/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:58
google.golang.org/grpc.(*Server).processStreamingRPC
	/home/builder/go/pkg/mod/google.golang.org/[email protected]/server.go:1540
google.golang.org/grpc.(*Server).handleStream
	/home/builder/go/pkg/mod/google.golang.org/[email protected]/server.go:1613
google.golang.org/grpc.(*Server).serveStreams.func1.2
	/home/builder/go/pkg/mod/google.golang.org/[email protected]/server.go:934
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1371
failed to find agent
github.com/percona/pmm-managed/services/agents.authenticate
	/home/builder/rpm/BUILD/pmm-managed-f581ced32bf5231e0bc296d19e005e90ead70d1f/src/github.com/percona/pmm-managed/services/agents/registry.go:218
github.com/percona/pmm-managed/services/agents.(*Registry).register.func1
	/home/builder/rpm/BUILD/pmm-managed-f581ced32bf5231e0bc296d19e005e90ead70d1f/src/github.com/percona/pmm-managed/services/agents/registry.go:163
gopkg.in/reform%2ev1.(*DB).InTransactionContext
	/home/builder/go/pkg/mod/gopkg.in/[email protected]/db.go:93
gopkg.in/reform%2ev1.(*DB).InTransaction
	/home/builder/go/pkg/mod/gopkg.in/[email protected]/db.go:74
github.com/percona/pmm-managed/services/agents.(*Registry).register
	/home/builder/rpm/BUILD/pmm-managed-f581ced32bf5231e0bc296d19e005e90ead70d1f/src/github.com/percona/pmm-managed/services/agents/registry.go:162
github.com/percona/pmm-managed/services/agents.(*Handler).Run
	/home/builder/rpm/BUILD/pmm-managed-f581ced32bf5231e0bc296d19e005e90ead70d1f/src/github.com/percona/pmm-managed/services/agents/handler.go:70
github.com/percona/pmm-managed/services/agents/grpc.(*agentServer).Connect
	/home/builder/rpm/BUILD/pmm-managed-f581ced32bf5231e0bc296d19e005e90ead70d1f/src/github.com/percona/pmm-managed/services/agents/grpc/agent_server.go:39
github.com/percona/pmm/api/agentpb._Agent_Connect_Handler
	/home/builder/go/pkg/mod/github.com/percona/[email protected]/api/agentpb/agent.pb.go:7212
github.com/grpc-ecosystem/go-grpc-middleware/validator.StreamServerInterceptor.func1
	/home/builder/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/validator/validator.go:41
github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1
	/home/builder/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49
github.com/percona/pmm-managed/utils/interceptors.StreamServiceEnabledInterceptor.func1
	/home/builder/rpm/BUILD/pmm-managed-f581ced32bf5231e0bc296d19e005e90ead70d1f/src/github.com/percona/pmm-managed/utils/interceptors/service.go:50
github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1
	/home/builder/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49
github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).StreamServerInterceptor.func1
	/home/builder/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/server_metrics.go:121
github.com/percona/pmm-managed/utils/interceptors.Stream.func1
	/home/builder/rpm/BUILD/pmm-managed-f581ced32bf5231e0bc296d19e005e90ead70d1f/src/github.com/percona/pmm-managed/utils/interceptors/interceptors.go:125
github.com/percona/pmm-managed/utils/interceptors.logRequest
	/home/builder/rpm/BUILD/pmm-managed-f581ced32bf5231e0bc296d19e005e90ead70d1f/src/github.com/percona/pmm-managed/utils/interceptors/interceptors.go:78
github.com/percona/pmm-managed/utils/interceptors.Stream
	/home/builder/rpm/BUILD/pmm-managed-f581ced32bf5231e0bc296d19e005e90ead70d1f/src/github.com/percona/pmm-managed/utils/interceptors/interceptors.go:122
github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1
	/home/builder/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49
github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1
	/home/builder/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:58
google.golang.org/grpc.(*Server).processStreamingRPC
	/home/builder/go/pkg/mod/google.golang.org/[email protected]/server.go:1540
google.golang.org/grpc.(*Server).handleStream
	/home/builder/go/pkg/mod/google.golang.org/[email protected]/server.go:1613
google.golang.org/grpc.(*Server).serveStreams.func1.2
	/home/builder/go/pkg/mod/google.golang.org/[email protected]/server.go:934
runtime.goexit
	/usr/local/go/src/runtime/asm_amd64.s:1371  agent_id=pmm-server request=e6be0ad2-f91f-11eb-bbcc-0242ac110002
INFO[2021-08-09T14:41:38.798+00:00] Migrating database...                         component=setup
WARN[2021-08-09T14:41:38.800+00:00] Failed to migrate database: failed to execute statement:
ALTER TABLE artifacts
      ADD COLUMN type VARCHAR NOT NULL CHECK (type <> '') DEFAULT 'on_demand',
      ADD COLUMN schedule_id VARCHAR: pq: column "type" of relation "artifacts" already exists.  component=setup
WARN[2021-08-09T14:41:38.800+00:00] Retrying in 2s.                               component=setup
INFO[2021-08-09T14:41:40.473+00:00] Starting RPC /server.Server/Readiness ...     request=e87ce20d-f91f-11eb-bbcc-0242ac110002
INFO[2021-08-09T14:41:40.477+00:00] RPC /server.Server/Readiness done in 3.816776ms.  request=e87ce20d-f91f-11eb-bbcc-0242ac110002
INFO[2021-08-09T14:41:40.800+00:00] Migrating database...                         component=setup
WARN[2021-08-09T14:41:40.801+00:00] Failed to migrate database: failed to execute statement:


       [[email protected] opt]# supervisorctl status
alertmanager                     RUNNING   pid 18119, uptime 0:01:47
clickhouse                       RUNNING   pid 17278, uptime 0:02:23
cron                             RUNNING   pid 18, uptime 8:19:55
dashboard-upgrade                EXITED    Aug 09 02:39 PM
dbaas-controller                 STOPPED   Not started
grafana                          RUNNING   pid 19171, uptime 0:00:34
nginx                            RUNNING   pid 15503, uptime 0:03:49
pmm-agent                        RUNNING   pid 18501, uptime 0:01:29
pmm-managed                      RUNNING   pid 17763, uptime 0:02:07
pmm-update-perform               EXITED    Aug 09 02:39 PM
postgresql                       RUNNING   pid 17994, uptime 0:01:54
prometheus                       STOPPED   Not started
qan-api2                         RUNNING   pid 18374, uptime 0:01:35
victoriametrics                  RUNNING   pid 19, uptime 8:19:55
vmalert                          RUNNING   pid 20, uptime 8:19:55

      Attachments

        Activity

          People

            nurlan.moldomurov Nurlan Moldomurov
            nailya.kutlubaeva Nailya Kutlubaeva
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Smart Checklist