Details
-
Bug
-
Status: Done
-
High
-
Resolution: Fixed
-
None
-
1
-
Yes
-
Yes
-
Yes
-
[obsolete] C/S Core
-
Description
User Impact: Broken Dashboards with no data and only Templating Error to display
While verifying the performance for PMM-Server and UI response with server monitoring 1000 nodes & services, we observed an issue with changing scrape Frequency, causing many dashboards to show templating error.
Template variable service failed cannot fetch data for "MinTimestamp=2022-05-23 20:21:36 +0000 UTC, MaxTimestamp=2022-05-24 08:21:36 +0000 UTC, TagFilters=[\n{Key=\"\", Value=\"up\", IsNegative: false, IsRegexp: false}\n]": search error after reading 0 data blocks: error when searching for tagFilters=[{__name__="up"}] on the time range [2022-05-23 20:21:36 +0000 UTC - 2022-05-24 08:21:36 +0000 UTC]: error when searching tsids: the number of matching timeseries exceeds 10000; either narrow down the search or increase -search.max* command-line flag values at vmselect
A similar error could be easily observed on MySQL Table Details dashboard too, it can be easily seen on a PMM-Server intance monitoring roughly 100 mysql services
STR:
1) Monitor 1000 apprx. nodes & service with PMM
2) Change Scrape Interval from Standard to Frequent
3) Open Home Dashboard, Node Summary etc
4) You will see the error with a limit on matching time-series with the given time range, it seems we need to increase the default VM parameters.