Uploaded image for project: 'Percona Monitoring and Management'
  1. Percona Monitoring and Management
  2. PMM-4124

Trying to Add 100 PS nodes to PMM-client fails with error: too many connections

    Details

    • Type: Bug
    • Status: Done
    • Priority: Medium
    • Resolution: Invalid
    • Affects Version/s: 2.0.0-alpha4
    • Fix Version/s: None
    • Component/s: PMM Client
    • Labels:
      None

      Description

      I have been trying to test the performance of PMM-server host and pmm-client host when users try to add 100 nodes for monitoring, started with Percona-Server. 

      Host for PMM-server and PMM-client both are different. 

      Steps to Reproduce the error. 

      1) Add 1 Percona Server node for monitoring using PS Docker container or any other deployment method. 

      2) Use the same node and add it to monitoring using a different name, keep adding the same node multiple times, with different names. 

      I created a simple script to add 100 nodes, with different names..each node is added after an interval of 30 seconds.

      Nodes keep on getting added until 46th Node and after that, we start getting too many connections error, and somewhere between 46-100 nodes, one or two nodes are successfully added. 

      I have provided logs for the failure, output of debug when trying to add the node. 

       

       

      {"error":"Error 1040: Too many connections","message":"Error 1040: Too many connections","code":9}
      0
      Result: <nil>                               
      Error: &my_sql.AddMySQLDefault{_statusCode:412, Payload:(*my_sql.AddMySQLDefaultBody)(0xc000388b10)}
      Error 1040: Too many connections
      

      PMM-agent log file is just too huge, let me know if that is also needed, Debug Log PMM-Client Too many connections error.log

      Error 1040: Too many connections\" source=\"exporter.go:110\"" agentID=/agent_id/9a128605-490b-42f6-9671-16380e21f2fb component=agent-process type=mysqld_exporter
      time="2019-05-25T09:33:49Z" level=info msg="time=\"2019-05-25T09:33:49Z\" level=error msg=\"Error scraping for collect.perf_schema.file_events: Error 1040: Too many connections\" source=\"exporter.go:110\"" agentID=/agent_id/c8bafab1-0aab-40b3-af78-5b64e9465818 component=agent-process type=mysqld_exporter
      

       

       

        Smart Checklist

          Attachments

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                puneet.kala Puneet Kala
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: