Some panels not visible when using long time intervals (e.g. 30 days)

Impact on the user:

• The user is unable to see some panels with long time interval selected

Expected Result:

• All panels rendered 

Given Result:

• error on panels 

Workaround:
N/a for a long period of the time interval or increase pmm-server resources 

 

Suggested implementation:

• add parameter search.maxConcurrentRequests to victoriametrics in supervisord in pmm-managed.
• this will inscrease default value. We need set this value probably at 32 and higher (can depends on number of graphs in node summary).
• flags documentation: https://hub.docker.com/r/victoriametrics/victoria-metrics/
• another flags, what we can try if they helps: search.maxQueueDuration, search.maxQueryDuration

How to test?

1. Mount pmm data volume from pmm-demo (ask IT guys)
2. go to PMM Nodes Summary dashboard 
3. select pmm-server
4. set time interval to 30 day, or bigger

Reproducible probably on slower instancies due to low default value for connections.

Can reproduce: https://pmmdemo.percona.com/graph/d/node-instance-summary/node-summary?orgId=1&refresh=1m&var-interval=$__auto_interval_interval&var-node_name=pmm-server&var-service_name=All&var-environment=All&var-cluster=All&var-replication_set=All&var-database=All&var-node_type=All&var-service_type=All&var-username=All&var-schema=All&var-node_id=pmm-server&from=now-5m&to=now

Cannot reproduce: http://3.135.248.193/graph/d/node-instance-summary/node-summary?orgId=1&refresh=1m&var-interval=$__auto_interval_interval&var-node_name=pmm-server&var-service_name=All&var-environment=All&var-cluster=All&var-replication_set=All&var-database=All&var-node_type=All&var-service_type=All&var-username=All&var-schema=All&var-node_id=pmm-server&from=now-5m&to=now

Locally cannot reproduce it at all

 

 

Original report:

 

Error text:

remoteAddr: "127.0.0.1:37826", X-Forwarded-For: "127.0.0.1, 127.0.0.1"; cannot handle more than 8 concurrent search requests during 10s; possible solutions: increase `-search.maxQueueDuration`; increase `-search.maxQueryDuration`; increase `-search.maxConcurrentRequests`; increase server capacity