Details
-
Bug
-
Status: Done
-
Medium
-
Resolution: Won't Fix
-
2.30.0
-
None
-
Yes
-
Yes
-
Yes
-
No
-
[obsolete] Server Integrations
Description
User impact:
DB cluster creation fails
Steps to reproduce:
Start pmm-server https://pmm.cd.percona.com/blue/organizations/jenkins/aws-staging-start/activity/ 2.30.0 RC with -e PERCONA_TEST_DBAAS_PMM_CLIENT=percona/pmm-client:xxx env variable
perconalab/pmm-server:2.30.0-rc
Start EKS cluster
Set wrong Public Address in PMM Settings
Register EKS cluster
Go to dbaas and create a db cluster
Actual result:
Pods in CrashLoopBackOff
INFO[2022-08-22T12:15:57.411+00:00] Updating PMM Server address from "18.188.176.15" to "18.188.176.15:443". component=main Checking local pmm-agent status... pmm-agent is not running. Registering pmm-agent on PMM Server... INFO[2022-08-22T12:15:57.411+00:00] Using /usr/local/percona/pmm2/exporters/node_exporter component=setup INFO[2022-08-22T12:15:57.411+00:00] Using /usr/local/percona/pmm2/exporters/mysqld_exporter component=setup INFO[2022-08-22T12:15:57.411+00:00] Using /usr/local/percona/pmm2/exporters/mongodb_exporter component=setup INFO[2022-08-22T12:15:57.411+00:00] Using /usr/local/percona/pmm2/exporters/postgres_exporter component=setup INFO[2022-08-22T12:15:57.411+00:00] Using /usr/local/percona/pmm2/exporters/proxysql_exporter component=setup INFO[2022-08-22T12:15:57.411+00:00] Using /usr/local/percona/pmm2/exporters/rds_exporter component=setup INFO[2022-08-22T12:15:57.411+00:00] Using /usr/local/percona/pmm2/exporters/azure_exporter component=setup INFO[2022-08-22T12:15:57.411+00:00] Window check connection time is 1.00 hour(s) INFO[2022-08-22T12:15:57.411+00:00] Using /usr/local/percona/pmm2/exporters/vmagent component=setup INFO[2022-08-22T12:15:57.411+00:00] Updating PMM Server address from "18.188.176.15" to "18.188.176.15:443". component=setup INFO[2022-08-22T12:15:57.411+00:00] Runner capacity set to 32. component=runner INFO[2022-08-22T12:15:57.412+00:00] Starting... component=client ERRO[2022-08-22T12:15:57.412+00:00] Agent ID is not provided, halting. component=client INFO[2022-08-22T12:15:57.412+00:00] Starting local API server on http://0.0.0.0:7777/ ... component=local-server/JSON INFO[2022-08-22T12:15:57.413+00:00] Started. component=local-server/JSON Failed to register pmm-agent on PMM Server: json: cannot unmarshal object into Go struct field RegisterNodeDefaultBody.details of type []*node.RegisterNodeDefaultBodyDetailsItems0. INFO[2022-08-22T12:15:57.454+00:00] 'pmm-admin setup' exited with 1 component=entrypoint INFO[2022-08-22T12:15:57.454+00:00] Restarting `pmm-admin setup` in 5 seconds because PMM_AGENT_SIDECAR is enabled... component=entrypoint INFO[2022-08-22T12:16:02.458+00:00] Starting 'pmm-admin setup'... component=entrypoint INFO[2022-08-22T12:16:02.471+00:00] Loading configuration file /usr/local/percona/pmm2/config/pmm-agent.yaml. component=setup INFO[2022-08-22T12:16:02.471+00:00] Using /usr/local/percona/pmm2/exporters/node_exporter component=setup INFO[2022-08-22T12:16:02.471+00:00] Using /usr/local/percona/pmm2/exporters/mysqld_exporter component=setup INFO[2022-08-22T12:16:02.471+00:00] Using /usr/local/percona/pmm2/exporters/mongodb_exporter component=setup INFO[2022-08-22T12:16:02.471+00:00] Using /usr/local/percona/pmm2/exporters/postgres_exporter component=setup INFO[2022-08-22T12:16:02.471+00:00] Using /usr/local/percona/pmm2/exporters/proxysql_exporter component=setup INFO[2022-08-22T12:16:02.471+00:00] Using /usr/local/percona/pmm2/exporters/rds_exporter component=setup Checking local pmm-agent status... INFO[2022-08-22T12:16:02.471+00:00] Using /usr/local/percona/pmm2/exporters/azure_exporter component=setup INFO[2022-08-22T12:16:02.471+00:00] Using /usr/local/percona/pmm2/exporters/vmagent component=setup INFO[2022-08-22T12:16:02.471+00:00] Updating PMM Server address from "18.188.176.15" to "18.188.176.15:443". component=setup pmm-agent is running. Registering pmm-agent on PMM Server... Failed to register pmm-agent on PMM Server: json: cannot unmarshal object into Go struct field RegisterNodeDefaultBody.details of type []*node.RegisterNodeDefaultBodyDetailsItems0. ERRO[2022-08-22T12:16:02.518+00:00] Can't get exit code for 'pmm-admin setup'. err: wait: no child processes component=entrypoint beata@OSR-LAP-045:~$ beata@OSR-LAP-045:~$ beata@OSR-LAP-045:~$ kubectl get pod --kubeconfig=/home/beata/.kube/my_eks_conf NAME READY STATUS RESTARTS AGE mysql-q2uxm5-haproxy-0 2/3 CrashLoopBackOff 10 (51s ago) 27m mysql-q2uxm5-pxc-0 1/2 CrashLoopBackOff 10 (48s ago) 27m percona-server-mongodb-operator-665cd69f9b-g74w5 1/1 Running 0 3d percona-xtradb-cluster-operator-84cf7fc56c-9mpvr 1/1 Running 4 (9d ago) 12d
Expected result:
DB cluster creation should complete
Workaround:
-
Details:
This works fine when running pmm-server locally + local minikube
The pods don't fail and pmm-agent keeps restarting as expected:
Checking local pmm-agent status... pmm-agent is running. Registering pmm-agent on PMM Server... Failed to register pmm-agent on PMM Server: Post "https://localhost:443/v1/management/Node/Register": dial tcp 127.0.0.1:443: connect: connection refused. ERRO[2022-08-22T12:20:55.055+00:00] Can't get exit code for 'pmm-admin setup'. err: wait: no child processes component=entrypoint INFO[2022-08-22T12:20:55.055+00:00] 'pmm-admin setup' exited with -1 component=entrypoint INFO[2022-08-22T12:20:55.055+00:00] Restarting `pmm-admin setup` in 5 seconds because PMM_AGENT_SIDECAR is enabled... component=entrypoint INFO[2022-08-22T12:21:00.059+00:00] Starting 'pmm-admin setup'... component=entrypoint INFO[2022-08-22T12:21:00.091+00:00] Loading configuration file /usr/local/percona/pmm2/config/pmm-agent.yaml. component=setup INFO[2022-08-22T12:21:00.092+00:00] Using /usr/local/percona/pmm2/exporters/node_exporter component=setup INFO[2022-08-22T12:21:00.092+00:00] Using /usr/local/percona/pmm2/exporters/mysqld_exporter component=setup INFO[2022-08-22T12:21:00.092+00:00] Using /usr/local/percona/pmm2/exporters/mongodb_exporter component=setup INFO[2022-08-22T12:21:00.092+00:00] Using /usr/local/percona/pmm2/exporters/postgres_exporter component=setup INFO[2022-08-22T12:21:00.092+00:00] Using /usr/local/percona/pmm2/exporters/proxysql_exporter component=setup INFO[2022-08-22T12:21:00.092+00:00] Using /usr/local/percona/pmm2/exporters/rds_exporter component=setup INFO[2022-08-22T12:21:00.092+00:00] Using /usr/local/percona/pmm2/exporters/azure_exporter component=setup INFO[2022-08-22T12:21:00.092+00:00] Using /usr/local/percona/pmm2/exporters/vmagent component=setup INFO[2022-08-22T12:21:00.092+00:00] Updating PMM Server address from "localhost" to "localhost:443". component=setup Checking local pmm-agent status... pmm-agent is running. Registering pmm-agent on PMM Server... Failed to register pmm-agent on PMM Server: Post "https://localhost:443/v1/management/Node/Register": dial tcp 127.0.0.1:443: connect: connection refused. ERRO[2022-08-22T12:21:00.097+00:00] Can't get exit code for 'pmm-admin setup'. err: waitid: no child processes component=entrypoint INFO[2022-08-22T12:21:00.097+00:00] 'pmm-admin setup' exited with -1 component=entrypoint INFO[2022-08-22T12:21:00.097+00:00] Restarting `pmm-admin setup` in 5 seconds because PMM_AGENT_SIDECAR is enabled... component=entrypoint beata@OSR-LAP-045:~$ beata@OSR-LAP-045:~$ beata@OSR-LAP-045:~$ beata@OSR-LAP-045:~$ beata@OSR-LAP-045:~$ kubectl get pod NAME READY STATUS RESTARTS AGE mysql-ldpeq2-haproxy-0 3/3 Running 0 27m mysql-ldpeq2-pxc-0 2/2 Running 0 27m percona-server-mongodb-operator-547c499bd8-khrxb 1/1 Running 0 30m percona-xtradb-cluster-operator-79949dc46d-qr8cj 1/1 Running 0 30m vmagent-pmm-vmagent-vm-operator-15135024108570456149-fb7f7dvkm8 2/2 Running 0 29m xb-cron-mysql-ldpeq2-pxc-backup-stora-202282212300-3qf7g-vn4br 0/1 Completed 0 8m44s