Details

    • Type: Bug
    • Status: Open
    • Priority: Medium
    • Resolution: Unresolved
    • Affects Version/s: 1.17.0
    • Fix Version/s: None
    • Component/s: PMM Client
    • Labels:

      Description

      Hi,

      I observe this:

      root@skullnuc:/tmp# time pmm-admin restart --all
      OK, restarted 3 services.

      real 1m35.096s
      user 0m0.046s
      sys 0m0.024s

      1 minute 30 seconds is excessive.. .especially considering restart happens
      when package is updated in which case it just looks like stall after:

      Setting up pmm-client (1.15.0-1.bionic) ...

      root@skullnuc:/tmp# pmm-admin list
      pmm-admin 1.15.0

      PMM Server | 10.11.13.140
      Client Name | skullnuc
      Client Address | 10.11.13.30
      Service Manager | linux-systemd

      -------------- --------- ----------- --------
      -------------------------------------------
      --------------------------------------------------------------------------------------
      SERVICE TYPE NAME LOCAL PORT RUNNING DATA SOURCE
      OPTIONS
      -------------- --------- ----------- --------
      -------------------------------------------
      --------------------------------------------------------------------------------------
      mysql:queries skullnuc - YES
      root:***@unix(/var/run/mysqld/mysqld.sock) query_source=slowlog,
      query_examples=true, slow_log_rotation=true, retain_slow_logs=1
      linux:metrics skullnuc 42000 YES -
      mysql:metrics skullnuc 42002 NO
      root:***@unix(/var/run/mysqld/mysqld.sock)

      Job name Scrape interval Scrape timeout Metrics path Scheme
      Target Labels Health
      processes-rocky 10s 10s /metrics http
      10.11.13.141:9256 instance="rocky" UP

      Note MySQL Exporter is not running.

      Note MySQL Server is also currently not running on this host

      After Fixing MySQL I have:

      root@skullnuc:/tmp# time pmm-admin restart --all
      OK, restarted 3 services.

      real 1m30.818s
      user 0m0.045s
      sys 0m0.029s

      (still the issue)

      In Exporter Logs I have:

      time="2018-10-15T01:44:53+01:00" level=fatal msg="failed reading ini file:
      open .my.cnf: no such file or directory" source="mysqld_exporter.go:339"
      time="2018-10-15T01:46:53+01:00" level=info msg="Starting mysqld_exporter
      (version=, branch=, revision=)" source="mysqld_exporter.go:331"
      time="2018-10-15T01:46:53+01:00" level=info msg="Build context
      (go=go1.10.1, user=, date=)" source="mysqld_exporter.go:332"
      time="2018-10-15T01:46:53+01:00" level=fatal msg="failed reading ini file:
      open .my.cnf: no such file or directory" source="mysqld_exporter.go:339"
      time="2018-10-15T01:48:53+01:00" level=info msg="Starting mysqld_exporter
      (version=, branch=, revision=)" source="mysqld_exporter.go:331"
      time="2018-10-15T01:48:53+01:00" level=info msg="Build context
      (go=go1.10.1, user=, date=)" source="mysqld_exporter.go:332"
      time="2018-10-15T01:48:53+01:00" level=fatal msg="failed reading ini file:
      open .my.cnf: no such file or directory" source="mysqld_exporter.go:339"

      The exporter is really NOT running when I start restart process

      Removing MySQL Metrics to see if it is the issue:

      root@skullnuc:/tmp# pmm-admin remove mysql:metrics
      OK, removed MySQL metrics skullnuc from monitorin

      root@skullnuc:/tmp# time pmm-admin restart --all
      OK, restarted 2 services.

      real 0m2.100s
      user 0m0.026s
      sys 0m0.031s

      So the problem IS related to validating dead exporter

      This is excessive as in this network configuration attempting connection
      to exporter ports instantly gives connection refused so check should be
      quick:

      [root@celenuc2 ~]# time telnet skullnuc 42002
      Trying 10.11.13.30...
      telnet: connect to address 10.11.13.30: Connection refused

      real 0m0.015s
      user 0m0.003s
      sys 0m0.009s

      Once I got exporter starting correctly (still not sure what was the
      problem)

      pmm-admin --restart is quick


      Peter Zaitsev, CEO, Percona

        Smart Checklist

          Attachments

            Issue Links

              Activity

                People

                • Assignee:
                  Unassigned
                  Reporter:
                  pz Peter Zaitsev
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  4 Start watching this issue

                  Dates

                  • Created:
                    Updated: