Uploaded image for project: 'Percona Monitoring and Management'
  1. Percona Monitoring and Management
  2. PMM-7312

Error when accessing Metrics data on Dashboards for large installations

Details

    • Bug
    • Status: Done
    • Medium
    • Resolution: Fixed
    • 2.13.0
    • 2.14.0
    • Grafana Dashboards
    • None
    • 1
    • 01 - Core
    • Yes
    • Yes
    • C/S Core

    Description

      Issue:  Using PMM v2.13.0, based in docker, to monitor ~200 servers with 2-3 services each. Accessing TimeSeries Information panels dashboards return the following error:

      "error when executing query=\"topk(10, count({__name__=~\\\".+\\\"}) by (__name__))\" for (time=1610117211741, step=300000): search error after reading 0 data blocks: error when searching for tagFilters=[{__name__=~\".+\"}] on the time range [2021-01-08 14:36:51.741 +0000 UTC - 2021-01-08 14:46:51.741 +0000 UTC]: error when searching tsids: the number of matching unique timeseries exceeds 1500000; either narrow down the search or increase -search.maxUniqueTimeseries"

       

      The issue can be seen for Dashboards,
      PMM Home page
      Nodes Overview
      MySQL Instances Overview

      Steps to Reproduce:

      The same issue can be seen for 200+ services. tested as follows,

      Install PMM_SERVER

      docker create -v /srv --name pmm-data percona/pmm-server:2.13.0 /bin/true
      docker run -d -p 80:80 -p 443:443 --volumes-from pmm-data --name pmm-server --restart always percona/pmm-server:2.13.0

      Configure PMM client

      pmm-admin config --server-insecure-tls --server-url=https://admin:[email protected]:443

      For testing rather than installing 200+ mysql, start few mysql servers and add them in pmm monitoring with the different service names.

      Add it at once:

      for i in {1..202} ; do pmm-admin add mysql nodelong_string_something_big_$i 127.0.0.1:3306 --username=msandbox --password=msandbox; done

      try to see time-series data for services on the dashboard (MySQL /MySQL Instances Overview)

      you will see "error when executing query=" 

       

      The same will happen for 200+ nodes and related dashboards. 

      Related  Issue: https://github.com/Percona-Lab/pmm-submodules/pull/1374

       

      Actual result: Grafana dashboard gives error while loading when monitoring a large number of nodes/services like 200 to 300.

       

      Expected Result:  Grafana dashboard should load without error when monitoring a large number of nodes/services like 200 to 300.

       

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              lalit.choudhary Lalit Choudhary
              Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - Not Specified
                  Not Specified
                  Logged:
                  Time Spent - 7 hours, 30 minutes
                  7h 30m

                  Smart Checklist