Uploaded image for project: 'Percona Monitoring and Management'
  1. Percona Monitoring and Management
  2. PMM-1483

Error removing one of the services: request canceled (Client.Timeout exceeded while awaiting headers)

    Details

    • Type: Bug
    • Status: Done
    • Priority: Low
    • Resolution: Fixed
    • Affects Version/s: 1.3.0
    • Fix Version/s: 1.3.2
    • Component/s: PMM Client
    • Labels:
      None
    • Story Points:
      1

      Description

      Hi,
      Here is the case.
      Running pmm test suite against PXC:

      First run:

      $ instance_t="pxc" instance_c="3" tap=1 stress=1 table_c=100 bash ~/percona-qa/pmm-tests/pmm-testsuite.sh
      .
      .
      Wipe clients
      Shutting down mysql instance (--socket=/tmp/PXC_NODE_3.sock)
      Shutting down mysql instance (--socket=/tmp/PXC_NODE_2.sock)
      Shutting down mysql instance (--socket=/tmp/PXC_NODE_1.sock)
      Removing all local pmm client instances
      
      $ sudo pmm-admin list
      pmm-admin 1.3.0
      
      PMM Server      | 10.10.6.202 (insecure SSL, password-protected)
      Client Name     | qaserver-02.ci.percona.com
      Client Address  | 10.10.6.202 
      Service Manager | linux-systemd
      
      No services under monitoring.
      

      Second run:

      $ instance_t="pxc" instance_c="3" tap=1 stress=1 table_c=100 bash ~/percona-qa/pmm-tests/pmm-testsuite.sh
      .
      .
      Wipe clients
      Shutting down mysql instance (--socket=/tmp/PXC_NODE_3.sock)
      Shutting down mysql instance (--socket=/tmp/PXC_NODE_2.sock)
      Shutting down mysql instance (--socket=/tmp/PXC_NODE_1.sock)
      Removing all local pmm client instances
      
      $ sudo pmm-admin list
      pmm-admin 1.3.0
      
      PMM Server      | 10.10.6.202 (insecure SSL, password-protected)
      Client Name     | qaserver-02.ci.percona.com
      Client Address  | 10.10.6.202 
      Service Manager | linux-systemd
      
      -------------- ----------- ----------- -------- ------------------------------------ ---------------------------------------------
      SERVICE TYPE   NAME        LOCAL PORT  RUNNING  DATA SOURCE                          OPTIONS                                      
      -------------- ----------- ----------- -------- ------------------------------------ ---------------------------------------------
      mysql:queries  PXC_NODE-1  -           YES      root:***@unix(/tmp/PXC_NODE_1.sock)  query_source=perfschema, query_examples=true 
      mysql:queries  PXC_NODE-2  -           YES      root:***@unix(/tmp/PXC_NODE_2.sock)  query_source=perfschema, query_examples=true 
      mysql:queries  PXC_NODE-3  -           YES      root:***@unix(/tmp/PXC_NODE_3.sock)  query_source=perfschema, query_examples=true
      

      Trying to remove manually:

      $ sudo pmm-admin rm --all
      Error removing one of the services: Put https://pmm:abc123@10.10.6.202/qan-api/agents/f76321191c5d4681430c90b1ce2a17d0/cmd: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
      
      $ sudo pmm-admin list
      pmm-admin 1.3.0
      
      PMM Server      | 10.10.6.202 (insecure SSL, password-protected)
      Client Name     | qaserver-02.ci.percona.com
      Client Address  | 10.10.6.202 
      Service Manager | linux-systemd
      
      -------------- ----------- ----------- -------- ------------------------------------ ---------------------------------------------
      SERVICE TYPE   NAME        LOCAL PORT  RUNNING  DATA SOURCE                          OPTIONS                                      
      -------------- ----------- ----------- -------- ------------------------------------ ---------------------------------------------
      mysql:queries  PXC_NODE-1  -           YES      root:***@unix(/tmp/PXC_NODE_1.sock)  query_source=perfschema, query_examples=true 
      mysql:queries  PXC_NODE-2  -           YES      root:***@unix(/tmp/PXC_NODE_2.sock)  query_source=perfschema, query_examples=true 
      mysql:queries  PXC_NODE-3  -           YES      root:***@unix(/tmp/PXC_NODE_3.sock)  query_source=perfschema, query_examples=true
      

      But it is not shown in check-network output:

      $ sudo pmm-admin check-network
      PMM Network Status
      
      Server Address | 10.10.6.202
      Client Address | 10.10.6.202 
      
      * System Time
      NTP Server (0.pool.ntp.org)         | 2017-09-15 08:49:38 +0000 GMT
      PMM Server                          | 2017-09-15 08:49:38 +0000 GMT
      PMM Client                          | 2017-09-15 08:49:38 +0000 GMT
      PMM Server Time Drift               | OK
      PMM Client Time Drift               | OK
      PMM Client to PMM Server Time Drift | OK
      
      * Connection: Client --> Server
      -------------------- -------      
      SERVER SERVICE       STATUS       
      -------------------- -------      
      Consul API           OK
      Prometheus API       OK
      Query Analytics API  OK
      
      Connection duration | 114.721┬Ás
      Request duration    | 12.602574ms
      Full round trip     | 12.717295ms
      
      
      * Connection: Client <-- Server
      ------------- ----- ------------------ ------- ---------- ---------
      SERVICE TYPE  NAME  REMOTE ENDPOINT    STATUS  HTTPS/TLS  PASSWORD 
      ------------- ----- ------------------ ------- ---------- ---------
      

      Also there is no orphaned instances as well:

      $ sudo pmm-admin repair
      No orphaned services found.
      

      This is complicated situation also make inconsistent test runs where the rm --all did not wipe all clients to have clean server.

        Smart Checklist

          Attachments

            Issue Links

              Activity

                People

                • Assignee:
                  nailya.kutlubaeva Nailya Kutlubaeva
                  Reporter:
                  shahriyar.rzayev Shahriyar Rzayev (Inactive)
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  3 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: