Uploaded image for project: 'Percona Monitoring and Management'
  1. Percona Monitoring and Management
  2. PMM-2621

pmm-server not able to collect mysql metrics

    XMLWordPrintable

    Details

    • Type: Admin & Maintenance Task
    • Status: New
    • Priority: Medium
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: PMM Client, PMM Server
    • Labels:

      Description

      Hello All,

      I have installed pmm-server on RHEL 7.4 through docker and pmm-clients in two RHEL 6.6 servers.
      I am able to collect linux metrics from clients, but not able to collect mysql metrics from clients.
      I have telnet from pmm-server to clients and from clients to pmm server.

      pmm-admin list output
      01:11:20 # pmm-admin list
      pmm-admin 1.11.0

      PMM Server | [server_ip]
      Client Name | [client]
      Client Address | [client_ip]
      Service Manager | linux-upstart

      -------------- ------------ ----------- -------- ----------------------------- --------------------------------------------------------------------------------------
      SERVICE TYPE NAME LOCAL PORT RUNNING DATA SOURCE OPTIONS 
      -------------- ------------ ----------- -------- ----------------------------- --------------------------------------------------------------------------------------
      mysql:queries [server] - YES root:***@tcp(127.0.0.1:3306) query_source=slowlog, query_examples=true, slow_log_rotation=true, retain_slow_logs=1
      linux:metrics [server] 42000 YES - 
      mysql:metrics [server] 42002 YES root:***@tcp(127.0.0.1:3306) tablestats=OFF 

      pmm-admin check-network output

      01:14:24 # pmm-admin check-network
      PMM Network Status

      Server Address | [server]:80
      Client Address | [server]

      • System Time
        PMM Server | 2018-06-11 06:16:23 +0000 GMT
        PMM Client | 2018-06-11 01:16:29 -0500 CDT
        PMM Client to PMM Server Time Drift | OK
      •  Connection: Client --> Server

      SERVER SERVICE STATUS

      Consul API OK
      Prometheus API OK
      Query Analytics API OK

      Connection duration | 507.004µs
      Request duration | 9.683055ms
      Full round trip | 10.190059ms

      • Connection: Client <-- Server

      SERVICE TYPE NAME REMOTE ENDPOINT STATUS HTTPS/TLS PASSWORD

      linux:metrics client xx.xx.xx.xx:42000 OK YES -
      mysql:metrics client xx.xx.xx.xx:42002 DOWN YES -

      When an endpoint is down it may indicate that the corresponding service is stopped (run 'pmm-admin list' to verify).
      If it's running, check out the logs /var/log/pmm-*.log

      When all endpoints are down but 'pmm-admin list' shows they are up and no errors in the logs,
      check the firewall settings whether this system allows incoming connections from server to address ort in question.

      Also you can check the endpoint status by the URL: http://[server]/prometheus/targets

      In prometheus dashboard state of mysql metrics is down, rest all are up.

      output of /var/log/pmm-mysql-metrics-42002.log

      time="2018-06-08T11:56:02-05:00" level=info msg="Starting mysqld_exporter (version=, branch=, revision=)" source="mysqld_exporter.go:292"
      time="2018-06-08T11:56:02-05:00" level=info msg="Build context (go=go1.9.4, user=, date=)" source="mysqld_exporter.go:293"
      time="2018-06-08T11:56:02-05:00" level=info msg="HTTPS/TLS is enabled" source="mysqld_exporter.go:362"
      time="2018-06-08T11:56:02-05:00" level=info msg="Enabled High Resolution scrapers:" source="mysqld_exporter.go:376"
      time="2018-06-08T11:56:02-05:00" level=info msg=" --collect.info_schema.innodb_metrics" source="mysqld_exporter.go:378"
      time="2018-06-08T11:56:02-05:00" level=info msg=" --collect.global_status" source="mysqld_exporter.go:378"
      time="2018-06-08T11:56:02-05:00" level=info msg="Enabled Medium Resolution scrapers:" source="mysqld_exporter.go:382"
      time="2018-06-08T11:56:02-05:00" level=info msg=" --collect.info_schema.processlist" source="mysqld_exporter.go:384"
      time="2018-06-08T11:56:02-05:00" level=info msg=" --collect.slave_status" source="mysqld_exporter.go:384"
      time="2018-06-08T11:56:02-05:00" level=info msg=" --collect.perf_schema.eventswaits" source="mysqld_exporter.go:384"
      time="2018-06-08T11:56:02-05:00" level=info msg=" --collect.perf_schema.file_events" source="mysqld_exporter.go:384"
      time="2018-06-08T11:56:02-05:00" level=info msg=" --collect.info_schema.query_response_time" source="mysqld_exporter.go:384"
      time="2018-06-08T11:56:02-05:00" level=info msg="Enabled Low Resolution scrapers:" source="mysqld_exporter.go:388"
      time="2018-06-08T11:56:02-05:00" level=info msg=" --collect.global_variables" source="mysqld_exporter.go:390"
      time="2018-06-08T11:56:02-05:00" level=info msg=" --collect.binlog_size" source="mysqld_exporter.go:390"
      time="2018-06-08T11:56:02-05:00" level=info msg=" --collect.info_schema.userstats" source="mysqld_exporter.go:390"
      time="2018-06-08T11:56:02-05:00" level=info msg="Listening on [server]:42002" source="mysqld_exporter.go:399"
      time="2018-06-08T11:56:33-05:00" level=info msg="Starting mysqld_exporter (version=, branch=, revision=)" source="mysqld_exporter.go:292"
      time="2018-06-08T11:56:33-05:00" level=info msg="Build context (go=go1.9.4, user=, date=)" source="mysqld_exporter.go:293"
      time="2018-06-08T11:56:33-05:00" level=info msg="HTTPS/TLS is enabled" source="mysqld_exporter.go:362"
      time="2018-06-08T11:56:33-05:00" level=info msg="Enabled High Resolution scrapers:" source="mysqld_exporter.go:376"
      time="2018-06-08T11:56:33-05:00" level=info msg=" --collect.info_schema.innodb_metrics" source="mysqld_exporter.go:378"
      time="2018-06-08T11:56:33-05:00" level=info msg=" --collect.global_status" source="mysqld_exporter.go:378"
      time="2018-06-08T11:56:33-05:00" level=info msg="Enabled Medium Resolution scrapers:" source="mysqld_exporter.go:382"
      time="2018-06-08T11:56:33-05:00" level=info msg=" --collect.info_schema.processlist" source="mysqld_exporter.go:384"
      time="2018-06-08T11:56:33-05:00" level=info msg=" --collect.slave_status" source="mysqld_exporter.go:384"
      time="2018-06-08T11:56:33-05:00" level=info msg=" --collect.perf_schema.eventswaits" source="mysqld_exporter.go:384"
      time="2018-06-08T11:56:33-05:00" level=info msg=" --collect.perf_schema.file_events" source="mysqld_exporter.go:384"
      time="2018-06-08T11:56:33-05:00" level=info msg=" --collect.info_schema.query_response_time" source="mysqld_exporter.go:384"
      time="2018-06-08T11:56:33-05:00" level=info msg="Enabled Low Resolution scrapers:" source="mysqld_exporter.go:388"
      time="2018-06-08T11:56:33-05:00" level=info msg=" --collect.global_variables" source="mysqld_exporter.go:390"
      time="2018-06-08T11:56:33-05:00" level=info msg=" --collect.binlog_size" source="mysqld_exporter.go:390"
      time="2018-06-08T11:56:33-05:00" level=info msg=" --collect.info_schema.userstats" source="mysqld_exporter.go:390"
      time="2018-06-08T11:56:33-05:00" level=info msg="Listening on [server]:42002" source="mysqld_exporter.go:399"
      2018/06/08 11:58:19 http: TLS handshake error from [server]:52196: tls: first record does not look like a TLS handshake
      2018/06/08 12:20:33 http: Accept error: accept tcp [server]:42002: accept4: too many open files; retrying in 5ms
      2018/06/08 12:20:33 http: Accept error: accept tcp [server]:42002: accept4: too many open files; retrying in 10ms
      2018/06/08 12:20:33 http: Accept error: accept tcp [server]:42002: accept4: too many open files; retrying in 20ms
      2018/06/08 12:20:33 http: Accept error: accept tcp [server]:42002: accept4: too many open files; retrying in 40ms
      2018/06/08 12:20:33 http: Accept error: accept tcp [server]:42002: accept4: too many open files; retrying in 80ms
      2018/06/08 12:20:33 http: Accept error: accept tcp [server]:42002: accept4: too many open files; retrying in 160ms
      2018/06/08 12:20:33 http: Accept error: accept tcp [server]:42002: accept4: too many open files; retrying in 320ms

      Telnet is working fine and OS metrics is working, only for mysql metrics it is giving above timeout error.

      curl command is working from docker container to pmm-client
      docker exec -ti pmm-server bash
      curl --insecure [https://xx.xx.xx.xx:42002
      <html>
      <head><title>MySQLd 3-in-1 exporter</title></head>
      <body>
      <h1>MySQL 3-in-1 exporter</h1>
      <li><a href="/metrics-hr">high-res metrics</a></li>
      <li><a href="/metrics-mr">medium-res metrics</a></li>
      <li><a href="/metrics-lr">low-res metrics</a></li>
      </body>
      </html>|https://10.2.21.xx:42002/]

       

      netstat output from client.

      01:54:07 # netstat | grep 42002
      tcp 1 0 client:42002 pmm-server:47866 CLOSE_WAIT
      tcp 1 0 client:42002 pmm-server:48302 CLOSE_WAIT
      tcp 1 0 client:42002 pmm-server:47996 CLOSE_WAIT
      tcp 0 0 client:42002 pmm-server:48122 CLOSE_WAIT
      tcp 1 0 client:42002 pmm-server:47958 CLOSE_WAIT
      tcp 1 0 client:42002 pmm-server:47840 CLOSE_WAIT
      tcp 1 0 client:42002 pmm-server:48040 CLOSE_WAIT
      tcp 1 0 client:42002 pmm-server:48280 CLOSE_WAIT
      tcp 0 0 client:42002 pmm-server:48242 CLOSE_WAIT
      tcp 1 0 client:42002 pmm-server:48404 CLOSE_WAIT
      tcp 1 0 client:42002 pmm-server:48324 CLOSE_WAIT
      tcp 0 0 client:42002 pmm-server:47956 CLOSE_WAIT
      tcp 1 0 client:42002 pmm-server:48348 CLOSE_WAIT
      tcp 0 0 client:42002 pmm-server:47818 CLOSE_WAIT
      tcp 1 0 client:42002 pmm-server:48146 CLOSE_WAIT
      tcp 0 0 client:42002 pmm-server:48628 ESTABLISHED
      tcp 1 0 client:42002 pmm-server:48474 CLOSE_WAIT
      tcp 1 0 client:42002 pmm-server:48240 CLOSE_WAIT
      tcp 1 0 client:42002 pmm-server:48450 CLOSE_WAIT
      tcp 1 0 client:42002 pmm-server:48602 CLOSE_WAIT
      tcp 1 0 client:42002 pmm-server:48220 CLOSE_WAIT

       

      Container to client telnet is also working.

       

      I installed mysql on pmm-server host and configured pmm-client , here mysql metrics are working fine.

      Please help me to solve this issue.

       

      Thanks,

      Jagannath

       

       

       

       

       

        Smart Checklist

          Attachments

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                Jagannath Jagannath M K
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated: