Uploaded image for project: 'Percona Monitoring and Management'
  1. Percona Monitoring and Management
  2. PMM-7167

Some panels not visible when using long time intervals (e.g. 30 days)

Details

    • 1
    • Yes
    • [obsolete] C/S Core

    Description

      Impact on the user:

      • The user is unable to see some panels with long time interval selected

      Expected Result:

      • All panels rendered 

      Given Result:

      • error on panels 

      Workaround:
      N/a for a long period of the time interval or increase pmm-server resources 

       

      Suggested implementation:

      • add parameter search.maxConcurrentRequests to victoriametrics in supervisord in pmm-managed.
      • this will inscrease default value. We need set this value probably at 32 and higher (can depends on number of graphs in node summary).
      • flags documentation: https://hub.docker.com/r/victoriametrics/victoria-metrics/
      • another flags, what we can try if they helps: search.maxQueueDuration, search.maxQueryDuration

      How to test?

      1. Mount pmm data volume from pmm-demo (ask IT guys)
      2. go to PMM Nodes Summary dashboard 
      3. select pmm-server
      4. set time interval to 30 day, or bigger

      Reproducible probably on slower instancies due to low default value for connections.

      Can reproduce: https://pmmdemo.percona.com/graph/d/node-instance-summary/node-summary?orgId=1&refresh=1m&var-interval=$__auto_interval_interval&var-node_name=pmm-server&var-service_name=All&var-environment=All&var-cluster=All&var-replication_set=All&var-database=All&var-node_type=All&var-service_type=All&var-username=All&var-schema=All&var-node_id=pmm-server&from=now-5m&to=now

      Cannot reproduce: http://3.135.248.193/graph/d/node-instance-summary/node-summary?orgId=1&refresh=1m&var-interval=$__auto_interval_interval&var-node_name=pmm-server&var-service_name=All&var-environment=All&var-cluster=All&var-replication_set=All&var-database=All&var-node_type=All&var-service_type=All&var-username=All&var-schema=All&var-node_id=pmm-server&from=now-5m&to=now

      Locally cannot reproduce it at all

       

       

      Original report:

       

      Error text:

      remoteAddr: "127.0.0.1:37826", X-Forwarded-For: "127.0.0.1, 127.0.0.1"; cannot handle more than 8 concurrent search requests during 10s; possible solutions: increase `-search.maxQueueDuration`; increase `-search.maxQueryDuration`; increase `-search.maxConcurrentRequests`; increase server capacity

       


      Attachments

        Activity

          People

            jiri.ctvrtka Jiří Čtvrtka
            roma.novikov Roma Novikov
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - Not Specified
                Not Specified
                Logged:
                Time Spent - 1 day, 15 minutes
                1d 15m

                Smart Checklist