Uploaded image for project: 'Percona Monitoring and Management'
  1. Percona Monitoring and Management
  2. PMM-1752

Pmm-client network failure with AWS AMI pmm-server.

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Priority: Medium
    • Resolution: Fixed
    • Affects Version/s: 1.4.0
    • Fix Version/s: 1.5.0
    • Component/s: None
    • Labels:
      None

      Description

      Steps which I performed
      1) Configured PMM server with latest image (ami-ebea6091)
      2) Started another aws instance and added PMM server info using pmm-client-1.5

      sudo pmm-admin config --server 34.231.122.152 --bind-address 172.30.6.186 --client-address 34.236.171.111 --server-user=ramesh --server-password=test --server-insecure-ssl
      sudo pmm-admin add mysql --user=root --socket=/tmp/PS_NODE_1.sock
      

      PMM server could not collect PS instance data due to network issue

      [admin@ip-172-30-6-186 pmmwork]$ sudo pmm-admin list
      pmm-admin 1.5.0
      
      PMM Server      | 34.231.122.152 (insecure SSL, password-protected)
      Client Name     | ip-172-30-6-186.ec2.internal
      Client Address  | 34.236.171.111 (172.30.6.186)
      Service Manager | linux-systemd
      
      -------------- ---------- ----------- -------- ----------------------------------- ---------------------------------------------
      SERVICE TYPE   NAME       LOCAL PORT  RUNNING  DATA SOURCE                         OPTIONS                                
      -------------- ---------- ----------- -------- ----------------------------------- ---------------------------------------------
      mysql:queries  PS_NODE-1  -           YES      root:***@unix(/tmp/PS_NODE_1.sock)  query_source=perfschema, query_examples=true
      linux:metrics  PS_NODE-1  42000       YES      -                                                                          
      mysql:metrics  PS_NODE-1  42002       YES      root:***@unix(/tmp/PS_NODE_1.sock)                                         
      [admin@ip-172-30-6-186 pmmwork]$ sudo pmm-admin check-network
      PMM Network Status
      
      Server Address | 34.231.122.152
      Client Address | 34.236.171.111 (172.30.6.186)
      
      * System Time
      NTP Server (0.pool.ntp.org)         | 2017-11-23 10:34:00 +0000 UTC
      PMM Server                          | 2017-11-23 10:34:00 +0000 GMT
      PMM Client                          | 2017-11-23 10:34:00 +0000 UTC
      PMM Server Time Drift               | OK
      PMM Client Time Drift               | OK
      PMM Client to PMM Server Time Drift | OK
      
      * Connection: Client --> Server
      -------------------- -------
      SERVER SERVICE       STATUS
      -------------------- -------
      Consul API           OK
      Prometheus API       OK
      Query Analytics API  OK
      
      Connection duration | 479.105┬Ás
      Request duration    | 19.6569ms
      Full round trip     | 20.136005ms
      
      
      * Connection: Client <-- Server
      -------------- ---------- ------------------------------------ ------- ---------- ---------
      SERVICE TYPE   NAME       REMOTE ENDPOINT                      STATUS  HTTPS/TLS  PASSWORD
      -------------- ---------- ------------------------------------ ------- ---------- ---------
      linux:metrics  PS_NODE-1  34.236.171.111-->172.30.6.186:42000  DOWN    YES        YES
      mysql:metrics  PS_NODE-1  34.236.171.111-->172.30.6.186:42002  DOWN    YES        YES
      
      When an endpoint is down it may indicate that the corresponding service is stopped (run 'pmm-admin list' to verify).
      If it's running, check out the logs /var/log/pmm-*.log
      
      When all endpoints are down but 'pmm-admin list' shows they are up and no errors in the logs,
      check the firewall settings whether this system allows incoming connections from server to address:port in question.
      
      Also you can check the endpoint status by the URL: https://34.231.122.152/prometheus/targets
      
      IMPORTANT: client and bind addresses are not the same which means you need to configure NAT/port forwarding to map them.
      
      [admin@ip-172-30-6-186 pmmwork]$
      

        Smart Checklist

          Attachments

            Activity

              People

              Assignee:
              Unassigned
              Reporter:
              ramesh.sivaraman Ramesh Sivaraman
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: