[PMM-10556] DBaaS: Failed to register pmm-agent on PMM Server - Percona JIRA

Details

Type: Bug
Status: Done
Priority: Medium
Resolution: Won't Fix
Affects Version/s: 2.30.0
Fix Version/s: None
Component/s: DBaaS controller
Labels:
- defect
- platform-stability
Environment:

https://18.188.176.157/graph/dbaas/dbclusters

Regression Issue:

Yes
Needs Review:
Yes
Needs QA:
Yes
Needs Doc:
No
Platform Team:
Server Integrations

Description

User impact:

DB cluster creation fails

Steps to reproduce:

Start pmm-server https://pmm.cd.percona.com/blue/organizations/jenkins/aws-staging-start/activity/ 2.30.0 RC with -e PERCONA_TEST_DBAAS_PMM_CLIENT=percona/pmm-client:xxx env variable

perconalab/pmm-server:2.30.0-rc

Start EKS cluster
Set wrong Public Address in PMM Settings
Register EKS cluster
Go to dbaas and create a db cluster

Actual result:

Pods in CrashLoopBackOff

INFO[2022-08-22T12:15:57.411+00:00] Updating PMM Server address from "18.188.176.15" to "18.188.176.15:443".  component=main
Checking local pmm-agent status...
pmm-agent is not running.
Registering pmm-agent on PMM Server...
INFO[2022-08-22T12:15:57.411+00:00] Using /usr/local/percona/pmm2/exporters/node_exporter  component=setup
INFO[2022-08-22T12:15:57.411+00:00] Using /usr/local/percona/pmm2/exporters/mysqld_exporter  component=setup
INFO[2022-08-22T12:15:57.411+00:00] Using /usr/local/percona/pmm2/exporters/mongodb_exporter  component=setup
INFO[2022-08-22T12:15:57.411+00:00] Using /usr/local/percona/pmm2/exporters/postgres_exporter  component=setup
INFO[2022-08-22T12:15:57.411+00:00] Using /usr/local/percona/pmm2/exporters/proxysql_exporter  component=setup
INFO[2022-08-22T12:15:57.411+00:00] Using /usr/local/percona/pmm2/exporters/rds_exporter  component=setup
INFO[2022-08-22T12:15:57.411+00:00] Using /usr/local/percona/pmm2/exporters/azure_exporter  component=setup
INFO[2022-08-22T12:15:57.411+00:00] Window check connection time is 1.00 hour(s) 
INFO[2022-08-22T12:15:57.411+00:00] Using /usr/local/percona/pmm2/exporters/vmagent  component=setup
INFO[2022-08-22T12:15:57.411+00:00] Updating PMM Server address from "18.188.176.15" to "18.188.176.15:443".  component=setup
INFO[2022-08-22T12:15:57.411+00:00] Runner capacity set to 32.                    component=runner
INFO[2022-08-22T12:15:57.412+00:00] Starting...                                   component=client
ERRO[2022-08-22T12:15:57.412+00:00] Agent ID is not provided, halting.            component=client
INFO[2022-08-22T12:15:57.412+00:00] Starting local API server on http://0.0.0.0:7777/ ...  component=local-server/JSON
INFO[2022-08-22T12:15:57.413+00:00] Started.                                      component=local-server/JSON
Failed to register pmm-agent on PMM Server: json: cannot unmarshal object into Go struct field RegisterNodeDefaultBody.details of type []*node.RegisterNodeDefaultBodyDetailsItems0.
INFO[2022-08-22T12:15:57.454+00:00] 'pmm-admin setup' exited with 1               component=entrypoint
INFO[2022-08-22T12:15:57.454+00:00] Restarting `pmm-admin setup` in 5 seconds because PMM_AGENT_SIDECAR is enabled...  component=entrypoint
INFO[2022-08-22T12:16:02.458+00:00] Starting 'pmm-admin setup'...                 component=entrypoint
INFO[2022-08-22T12:16:02.471+00:00] Loading configuration file /usr/local/percona/pmm2/config/pmm-agent.yaml.  component=setup
INFO[2022-08-22T12:16:02.471+00:00] Using /usr/local/percona/pmm2/exporters/node_exporter  component=setup
INFO[2022-08-22T12:16:02.471+00:00] Using /usr/local/percona/pmm2/exporters/mysqld_exporter  component=setup
INFO[2022-08-22T12:16:02.471+00:00] Using /usr/local/percona/pmm2/exporters/mongodb_exporter  component=setup
INFO[2022-08-22T12:16:02.471+00:00] Using /usr/local/percona/pmm2/exporters/postgres_exporter  component=setup
INFO[2022-08-22T12:16:02.471+00:00] Using /usr/local/percona/pmm2/exporters/proxysql_exporter  component=setup
INFO[2022-08-22T12:16:02.471+00:00] Using /usr/local/percona/pmm2/exporters/rds_exporter  component=setup
Checking local pmm-agent status...
INFO[2022-08-22T12:16:02.471+00:00] Using /usr/local/percona/pmm2/exporters/azure_exporter  component=setup
INFO[2022-08-22T12:16:02.471+00:00] Using /usr/local/percona/pmm2/exporters/vmagent  component=setup
INFO[2022-08-22T12:16:02.471+00:00] Updating PMM Server address from "18.188.176.15" to "18.188.176.15:443".  component=setup
pmm-agent is running.
Registering pmm-agent on PMM Server...
Failed to register pmm-agent on PMM Server: json: cannot unmarshal object into Go struct field RegisterNodeDefaultBody.details of type []*node.RegisterNodeDefaultBodyDetailsItems0.
ERRO[2022-08-22T12:16:02.518+00:00] Can't get exit code for 'pmm-admin setup'. err: wait: no child processes  component=entrypoint
[email protected]:~$ 
[email protected]:~$ 
[email protected]:~$ kubectl get pod --kubeconfig=/home/beata/.kube/my_eks_conf
NAME                                                              READY   STATUS             RESTARTS       AGE
mysql-q2uxm5-haproxy-0                                            2/3     CrashLoopBackOff   10 (51s ago)   27m
mysql-q2uxm5-pxc-0                                                1/2     CrashLoopBackOff   10 (48s ago)   27m
percona-server-mongodb-operator-665cd69f9b-g74w5                  1/1     Running            0              3d
percona-xtradb-cluster-operator-84cf7fc56c-9mpvr                  1/1     Running            4 (9d ago)     12d

Expected result:

DB cluster creation should complete

Workaround:

-

Details:

This works fine when running pmm-server locally + local minikube
The pods don't fail and pmm-agent keeps restarting as expected:

Checking local pmm-agent status...
pmm-agent is running.
Registering pmm-agent on PMM Server...
Failed to register pmm-agent on PMM Server: Post "https://localhost:443/v1/management/Node/Register": dial tcp 127.0.0.1:443: connect: connection refused.
ERRO[2022-08-22T12:20:55.055+00:00] Can't get exit code for 'pmm-admin setup'. err: wait: no child processes  component=entrypoint
INFO[2022-08-22T12:20:55.055+00:00] 'pmm-admin setup' exited with -1              component=entrypoint
INFO[2022-08-22T12:20:55.055+00:00] Restarting `pmm-admin setup` in 5 seconds because PMM_AGENT_SIDECAR is enabled...  component=entrypoint
INFO[2022-08-22T12:21:00.059+00:00] Starting 'pmm-admin setup'...                 component=entrypoint
INFO[2022-08-22T12:21:00.091+00:00] Loading configuration file /usr/local/percona/pmm2/config/pmm-agent.yaml.  component=setup
INFO[2022-08-22T12:21:00.092+00:00] Using /usr/local/percona/pmm2/exporters/node_exporter  component=setup
INFO[2022-08-22T12:21:00.092+00:00] Using /usr/local/percona/pmm2/exporters/mysqld_exporter  component=setup
INFO[2022-08-22T12:21:00.092+00:00] Using /usr/local/percona/pmm2/exporters/mongodb_exporter  component=setup
INFO[2022-08-22T12:21:00.092+00:00] Using /usr/local/percona/pmm2/exporters/postgres_exporter  component=setup
INFO[2022-08-22T12:21:00.092+00:00] Using /usr/local/percona/pmm2/exporters/proxysql_exporter  component=setup
INFO[2022-08-22T12:21:00.092+00:00] Using /usr/local/percona/pmm2/exporters/rds_exporter  component=setup
INFO[2022-08-22T12:21:00.092+00:00] Using /usr/local/percona/pmm2/exporters/azure_exporter  component=setup
INFO[2022-08-22T12:21:00.092+00:00] Using /usr/local/percona/pmm2/exporters/vmagent  component=setup
INFO[2022-08-22T12:21:00.092+00:00] Updating PMM Server address from "localhost" to "localhost:443".  component=setup
Checking local pmm-agent status...
pmm-agent is running.
Registering pmm-agent on PMM Server...
Failed to register pmm-agent on PMM Server: Post "https://localhost:443/v1/management/Node/Register": dial tcp 127.0.0.1:443: connect: connection refused.
ERRO[2022-08-22T12:21:00.097+00:00] Can't get exit code for 'pmm-admin setup'. err: waitid: no child processes  component=entrypoint
INFO[2022-08-22T12:21:00.097+00:00] 'pmm-admin setup' exited with -1              component=entrypoint
INFO[2022-08-22T12:21:00.097+00:00] Restarting `pmm-admin setup` in 5 seconds because PMM_AGENT_SIDECAR is enabled...  component=entrypoint
[email protected]:~$ 
[email protected]:~$ 
[email protected]:~$ 
[email protected]:~$ 
[email protected]:~$ kubectl get pod
NAME                                                              READY   STATUS      RESTARTS   AGE
mysql-ldpeq2-haproxy-0                                            3/3     Running     0          27m
mysql-ldpeq2-pxc-0                                                2/2     Running     0          27m
percona-server-mongodb-operator-547c499bd8-khrxb                  1/1     Running     0          30m
percona-xtradb-cluster-operator-79949dc46d-qr8cj                  1/1     Running     0          30m
vmagent-pmm-vmagent-vm-operator-15135024108570456149-fb7f7dvkm8   2/2     Running     0          29m
xb-cron-mysql-ldpeq2-pxc-backup-stora-202282212300-3qf7g-vn4br    0/1     Completed   0          8m44s

DBaaS: Failed to register pmm-agent on PMM Server

Details

Description

Attachments

Activity

People

Dates

Smart Checklist