Details
-
Bug
-
Status: Done
-
Medium
-
Resolution: Fixed
-
None
-
None
-
1
-
Yes
-
[obsolete] C/S Core
Description
Impact on the user:
- The user is unable to see some panels with long time interval selected
Expected Result:
- All panels rendered
Given Result:
- error on panels
Workaround:
N/a for a long period of the time interval or increase pmm-server resources
Suggested implementation:
- add parameter search.maxConcurrentRequests to victoriametrics in supervisord in pmm-managed.
- this will inscrease default value. We need set this value probably at 32 and higher (can depends on number of graphs in node summary).
- flags documentation: https://hub.docker.com/r/victoriametrics/victoria-metrics/
- another flags, what we can try if they helps: search.maxQueueDuration, search.maxQueryDuration
How to test?
- Mount pmm data volume from pmm-demo (ask IT guys)
- go to PMM Nodes Summary dashboard
- select pmm-server
- set time interval to 30 day, or bigger
Reproducible probably on slower instancies due to low default value for connections.
Locally cannot reproduce it at all
Original report:
Error text:
remoteAddr: "127.0.0.1:37826", X-Forwarded-For: "127.0.0.1, 127.0.0.1"; cannot handle more than 8 concurrent search requests during 10s; possible solutions: increase `-search.maxQueueDuration`; increase `-search.maxQueryDuration`; increase `-search.maxConcurrentRequests`; increase server capacity