Uploaded image for project: 'Percona Monitoring and Management'
  1. Percona Monitoring and Management
  2. PMM-10556

DBaaS: Failed to register pmm-agent on PMM Server

Details

    • Yes
    • Yes
    • Yes
    • No
    • Server Integrations

    Description

      User impact:

      DB cluster creation fails

      Steps to reproduce:

      Start pmm-server https://pmm.cd.percona.com/blue/organizations/jenkins/aws-staging-start/activity/ 2.30.0 RC with -e PERCONA_TEST_DBAAS_PMM_CLIENT=percona/pmm-client:xxx env variable

      perconalab/pmm-server:2.30.0-rc

      Start EKS cluster
      Set wrong Public Address in PMM Settings
      Register EKS cluster
      Go to dbaas and create a db cluster

      Actual result:

      Pods in CrashLoopBackOff

      INFO[2022-08-22T12:15:57.411+00:00] Updating PMM Server address from "18.188.176.15" to "18.188.176.15:443".  component=main
      Checking local pmm-agent status...
      pmm-agent is not running.
      Registering pmm-agent on PMM Server...
      INFO[2022-08-22T12:15:57.411+00:00] Using /usr/local/percona/pmm2/exporters/node_exporter  component=setup
      INFO[2022-08-22T12:15:57.411+00:00] Using /usr/local/percona/pmm2/exporters/mysqld_exporter  component=setup
      INFO[2022-08-22T12:15:57.411+00:00] Using /usr/local/percona/pmm2/exporters/mongodb_exporter  component=setup
      INFO[2022-08-22T12:15:57.411+00:00] Using /usr/local/percona/pmm2/exporters/postgres_exporter  component=setup
      INFO[2022-08-22T12:15:57.411+00:00] Using /usr/local/percona/pmm2/exporters/proxysql_exporter  component=setup
      INFO[2022-08-22T12:15:57.411+00:00] Using /usr/local/percona/pmm2/exporters/rds_exporter  component=setup
      INFO[2022-08-22T12:15:57.411+00:00] Using /usr/local/percona/pmm2/exporters/azure_exporter  component=setup
      INFO[2022-08-22T12:15:57.411+00:00] Window check connection time is 1.00 hour(s) 
      INFO[2022-08-22T12:15:57.411+00:00] Using /usr/local/percona/pmm2/exporters/vmagent  component=setup
      INFO[2022-08-22T12:15:57.411+00:00] Updating PMM Server address from "18.188.176.15" to "18.188.176.15:443".  component=setup
      INFO[2022-08-22T12:15:57.411+00:00] Runner capacity set to 32.                    component=runner
      INFO[2022-08-22T12:15:57.412+00:00] Starting...                                   component=client
      ERRO[2022-08-22T12:15:57.412+00:00] Agent ID is not provided, halting.            component=client
      INFO[2022-08-22T12:15:57.412+00:00] Starting local API server on http://0.0.0.0:7777/ ...  component=local-server/JSON
      INFO[2022-08-22T12:15:57.413+00:00] Started.                                      component=local-server/JSON
      Failed to register pmm-agent on PMM Server: json: cannot unmarshal object into Go struct field RegisterNodeDefaultBody.details of type []*node.RegisterNodeDefaultBodyDetailsItems0.
      INFO[2022-08-22T12:15:57.454+00:00] 'pmm-admin setup' exited with 1               component=entrypoint
      INFO[2022-08-22T12:15:57.454+00:00] Restarting `pmm-admin setup` in 5 seconds because PMM_AGENT_SIDECAR is enabled...  component=entrypoint
      INFO[2022-08-22T12:16:02.458+00:00] Starting 'pmm-admin setup'...                 component=entrypoint
      INFO[2022-08-22T12:16:02.471+00:00] Loading configuration file /usr/local/percona/pmm2/config/pmm-agent.yaml.  component=setup
      INFO[2022-08-22T12:16:02.471+00:00] Using /usr/local/percona/pmm2/exporters/node_exporter  component=setup
      INFO[2022-08-22T12:16:02.471+00:00] Using /usr/local/percona/pmm2/exporters/mysqld_exporter  component=setup
      INFO[2022-08-22T12:16:02.471+00:00] Using /usr/local/percona/pmm2/exporters/mongodb_exporter  component=setup
      INFO[2022-08-22T12:16:02.471+00:00] Using /usr/local/percona/pmm2/exporters/postgres_exporter  component=setup
      INFO[2022-08-22T12:16:02.471+00:00] Using /usr/local/percona/pmm2/exporters/proxysql_exporter  component=setup
      INFO[2022-08-22T12:16:02.471+00:00] Using /usr/local/percona/pmm2/exporters/rds_exporter  component=setup
      Checking local pmm-agent status...
      INFO[2022-08-22T12:16:02.471+00:00] Using /usr/local/percona/pmm2/exporters/azure_exporter  component=setup
      INFO[2022-08-22T12:16:02.471+00:00] Using /usr/local/percona/pmm2/exporters/vmagent  component=setup
      INFO[2022-08-22T12:16:02.471+00:00] Updating PMM Server address from "18.188.176.15" to "18.188.176.15:443".  component=setup
      pmm-agent is running.
      Registering pmm-agent on PMM Server...
      Failed to register pmm-agent on PMM Server: json: cannot unmarshal object into Go struct field RegisterNodeDefaultBody.details of type []*node.RegisterNodeDefaultBodyDetailsItems0.
      ERRO[2022-08-22T12:16:02.518+00:00] Can't get exit code for 'pmm-admin setup'. err: wait: no child processes  component=entrypoint
      [email protected]:~$ 
      [email protected]:~$ 
      [email protected]:~$ kubectl get pod --kubeconfig=/home/beata/.kube/my_eks_conf
      NAME                                                              READY   STATUS             RESTARTS       AGE
      mysql-q2uxm5-haproxy-0                                            2/3     CrashLoopBackOff   10 (51s ago)   27m
      mysql-q2uxm5-pxc-0                                                1/2     CrashLoopBackOff   10 (48s ago)   27m
      percona-server-mongodb-operator-665cd69f9b-g74w5                  1/1     Running            0              3d
      percona-xtradb-cluster-operator-84cf7fc56c-9mpvr                  1/1     Running            4 (9d ago)     12d 

      Expected result:

      DB cluster creation should complete

      Workaround:

      -

      Details:

      This works fine when running pmm-server locally + local minikube
      The pods don't fail and pmm-agent keeps restarting as expected:

      Checking local pmm-agent status...
      pmm-agent is running.
      Registering pmm-agent on PMM Server...
      Failed to register pmm-agent on PMM Server: Post "https://localhost:443/v1/management/Node/Register": dial tcp 127.0.0.1:443: connect: connection refused.
      ERRO[2022-08-22T12:20:55.055+00:00] Can't get exit code for 'pmm-admin setup'. err: wait: no child processes  component=entrypoint
      INFO[2022-08-22T12:20:55.055+00:00] 'pmm-admin setup' exited with -1              component=entrypoint
      INFO[2022-08-22T12:20:55.055+00:00] Restarting `pmm-admin setup` in 5 seconds because PMM_AGENT_SIDECAR is enabled...  component=entrypoint
      INFO[2022-08-22T12:21:00.059+00:00] Starting 'pmm-admin setup'...                 component=entrypoint
      INFO[2022-08-22T12:21:00.091+00:00] Loading configuration file /usr/local/percona/pmm2/config/pmm-agent.yaml.  component=setup
      INFO[2022-08-22T12:21:00.092+00:00] Using /usr/local/percona/pmm2/exporters/node_exporter  component=setup
      INFO[2022-08-22T12:21:00.092+00:00] Using /usr/local/percona/pmm2/exporters/mysqld_exporter  component=setup
      INFO[2022-08-22T12:21:00.092+00:00] Using /usr/local/percona/pmm2/exporters/mongodb_exporter  component=setup
      INFO[2022-08-22T12:21:00.092+00:00] Using /usr/local/percona/pmm2/exporters/postgres_exporter  component=setup
      INFO[2022-08-22T12:21:00.092+00:00] Using /usr/local/percona/pmm2/exporters/proxysql_exporter  component=setup
      INFO[2022-08-22T12:21:00.092+00:00] Using /usr/local/percona/pmm2/exporters/rds_exporter  component=setup
      INFO[2022-08-22T12:21:00.092+00:00] Using /usr/local/percona/pmm2/exporters/azure_exporter  component=setup
      INFO[2022-08-22T12:21:00.092+00:00] Using /usr/local/percona/pmm2/exporters/vmagent  component=setup
      INFO[2022-08-22T12:21:00.092+00:00] Updating PMM Server address from "localhost" to "localhost:443".  component=setup
      Checking local pmm-agent status...
      pmm-agent is running.
      Registering pmm-agent on PMM Server...
      Failed to register pmm-agent on PMM Server: Post "https://localhost:443/v1/management/Node/Register": dial tcp 127.0.0.1:443: connect: connection refused.
      ERRO[2022-08-22T12:21:00.097+00:00] Can't get exit code for 'pmm-admin setup'. err: waitid: no child processes  component=entrypoint
      INFO[2022-08-22T12:21:00.097+00:00] 'pmm-admin setup' exited with -1              component=entrypoint
      INFO[2022-08-22T12:21:00.097+00:00] Restarting `pmm-admin setup` in 5 seconds because PMM_AGENT_SIDECAR is enabled...  component=entrypoint
      [email protected]:~$ 
      [email protected]:~$ 
      [email protected]:~$ 
      [email protected]:~$ 
      [email protected]:~$ kubectl get pod
      NAME                                                              READY   STATUS      RESTARTS   AGE
      mysql-ldpeq2-haproxy-0                                            3/3     Running     0          27m
      mysql-ldpeq2-pxc-0                                                2/2     Running     0          27m
      percona-server-mongodb-operator-547c499bd8-khrxb                  1/1     Running     0          30m
      percona-xtradb-cluster-operator-79949dc46d-qr8cj                  1/1     Running     0          30m
      vmagent-pmm-vmagent-vm-operator-15135024108570456149-fb7f7dvkm8   2/2     Running     0          29m
      xb-cron-mysql-ldpeq2-pxc-backup-stora-202282212300-3qf7g-vn4br    0/1     Completed   0          8m44s 

      Attachments

        Activity

          People

            beata.handzelova Beata Handzelova
            beata.handzelova Beata Handzelova
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Smart Checklist