Uploaded image for project: 'Percona Monitoring and Management'
  1. Percona Monitoring and Management
  2. PMM-10065

PMM-Server monitoring 500+ nodes causes Templating error on multiple Dashboards, due to VM config defaults

Details

    • 1
    • Yes
    • Yes
    • Yes
    • C/S Core
    • Hide

      Test at instance 500+ nodes with differences look behind window.

      Show
      Test at instance 500+ nodes with differences look behind window.

    Description

      User Impact: Broken Dashboards with no data and only Templating Error to display

      While verifying the performance for PMM-Server and UI response with server monitoring 1000 nodes & services, we observed an issue with changing scrape Frequency, causing many dashboards to show templating error.

      Template variable service failed cannot fetch data for "MinTimestamp=2022-05-23 20:21:36 +0000 UTC, MaxTimestamp=2022-05-24 08:21:36 +0000 UTC, TagFilters=[\n{Key=\"\", Value=\"up\", IsNegative: false, IsRegexp: false}\n]": search error after reading 0 data blocks: error when searching for tagFilters=[{__name__="up"}] on the time range [2022-05-23 20:21:36 +0000 UTC - 2022-05-24 08:21:36 +0000 UTC]: error when searching tsids: the number of matching timeseries exceeds 10000; either narrow down the search or increase -search.max* command-line flag values at vmselect 

      A similar error could be easily observed on MySQL Table Details dashboard too, it can be easily seen on a PMM-Server intance monitoring roughly 100 mysql services

       

      STR:
      1) Monitor 1000 apprx. nodes & service with PMM

      2) Change Scrape Interval from Standard to Frequent

      3) Open Home Dashboard, Node Summary etc

      4) You will see the error with a limit on matching time-series with the given time range, it seems we need to increase the default VM parameters. 

      Attachments

        Issue Links

          Activity

            People

              anton.bystrov Anton Bystrov
              puneet.kala Puneet Kala
              Votes:
              1 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Smart Checklist