Uploaded image for project: 'Percona Monitoring and Management'
  1. Percona Monitoring and Management
  2. PMM-8090

Multi-request protection breaks metrics gathering

Details

    • [obsolete] C/S Core

    Description

      PMM has insurance when it is scraping metrics that does not allow to run new scrape task while another is still in progress. However, it sometimes does not allow to run independent metrics.

      How to Repeat

      Create a custom query with high resolution which will take 1 second to finish:

      bug_overlap:
              query: "select sleep(1)+1 as bug_overlap_hr from dual;"
              metrics:
                      - bug_overlap_hr:
                              usage: "GAUGE"
                              description: "Testing overlap"
      

      Wait a couple of minutes and find that auto_increment.columns metrics is also broken.

      Expected (because collection is buggy):

      9 Too Many Requests" source="mysqld_exporter.go:268"  agentID=/agent_id/4351c48b-5b81-4137-a65d-f78a6e478e17 component=agent-process type=mysqld_exporter
      May 13 01:58:38 Delly-7390 pmm-agent[3615156]: INFO[2021-05-13T01:58:38.380+03:00] 2021-05-12T22:58:38.379Z        error        VictoriaMetrics/lib/promscrape/scrapework.go:231        error when scraping "http://127.0.0.1:42000/metrics?collect%5B%5D=custom_query.mr&collect%5B%5D=engine_innodb_status&collect%5B%5D=info_schema.innodb_cmp&collect%5B%5D=info_schema.innodb_cmpmem&collect%5B%5D=info_schema.processlist&collect%5B%5D=info_schema.query_response_time&collect%5B%5D=perf_schema.eventswaits&collect%5B%5D=perf_schema.file_events&collect%5B%5D=perf_schema.tablelocks&collect%5B%5D=slave_status" from job "mysqld_exporter_agent_id_4351c48b-5b81-4137-a65d-f78a6e478e17_mr-10s" with labels {agent_id="/agent_id/4351c48b-5b81-4137-a65d-f78a6e478e17",agent_type="mysqld_exporter",instance="/agent_id/4351c48b-5b81-4137-a65d-f78a6e478e17",job="mysqld_exporter_agent_id_4351c48b-5b81-4137-a65d-f78a6e478e17_mr-10s",machine_id="/machine_id/450baf3f4c2947f18a9dc3a134becf74",node_id="/node_id/dcbb71cc-111f-4ca0-92da-09bc63728989",node_name="Delly-7390",node_type="generic",service_id="/service_id/f3b97579-221e-4e37-a9c7-ad11be6b18fd",service_name="Delly-7390-mysql",service_type="mysql"}: unexpected status code returned when scraping "http://127.0.0.1:42000/metrics?collect%5B%5D=custom_query.mr&collect%5B%5D=engine_innodb_status&collect%5B%5D=info_schema.innodb_cmp&collect%5B%5D=info_schema.innodb_cmpmem&collect%5B%5D=info_schema.processlist&collect%5B%5D=info_schema.query_response_time&collect%5B%5D=perf_schema.eventswaits&collect%5B%5D=perf_schema.file_events&collect%5B%5D=perf_schema.tablelocks&collect%5B%5D=slave_status": 429; expecting 200; response body: "429 Too Many Requests\n"  agentID=/agent_id/f3d8558b-c6f8-4e15-8d9b-40df654feb71 component=agent-process type=vm_agent
      

      Not expected (unrelated metric):

      9 Too Many Requests" source="mysqld_exporter.go:268"  agentID=/agent_id/4351c48b-5b81-4137-a65d-f78a6e478e17 component=agent-process type=mysqld_exporter
      May 13 01:58:38 Delly-7390 pmm-agent[3615156]: INFO[2021-05-13T01:58:38.703+03:00] 2021-05-12T22:58:38.703Z        error        VictoriaMetrics/lib/promscrape/scrapework.go:231        error when scraping "http://127.0.0.1:42000/metrics?collect%5B%5D=auto_increment.columns&collect%5B%5D=binlog_size&collect%5B%5D=custom_query.lr&collect%5B%5D=engine_tokudb_status&collect%5B%5D=global_variables&collect%5B%5D=heartbeat&collect%5B%5D=info_schema.clientstats&collect%5B%5D=info_schema.innodb_tablespaces&collect%5B%5D=info_schema.tables&collect%5B%5D=info_schema.tablestats&collect%5B%5D=info_schema.userstats&collect%5B%5D=perf_schema.eventsstatements&collect%5B%5D=perf_schema.file_instances&collect%5B%5D=perf_schema.indexiowaits&collect%5B%5D=perf_schema.tableiowaits" from job "mysqld_exporter_agent_id_4351c48b-5b81-4137-a65d-f78a6e478e17_lr-1m0s" with labels {agent_id="/agent_id/4351c48b-5b81-4137-a65d-f78a6e478e17",agent_type="mysqld_exporter",instance="/agent_id/4351c48b-5b81-4137-a65d-f78a6e478e17",job="mysqld_exporter_agent_id_4351c48b-5b81-4137-a65d-f78a6e478e17_lr-1m0s",machine_id="/machine_id/450baf3f4c2947f18a9dc3a134becf74",node_id="/node_id/dcbb71cc-111f-4ca0-92da-09bc63728989",node_name="Delly-7390",node_type="generic",service_id="/service_id/f3b97579-221e-4e37-a9c7-ad11be6b18fd",service_name="Delly-7390-mysql",service_type="mysql"}: unexpected status code returned when scraping "http://127.0.0.1:42000/metrics?collect%5B%5D=auto_increment.columns&collect%5B%5D=binlog_size&collect%5B%5D=custom_query.lr&collect%5B%5D=engine_tokudb_status&collect%5B%5D=global_variables&collect%5B%5D=heartbeat&collect%5B%5D=info_schema.clientstats&collect%5B%5D=info_schema.innodb_tablespaces&collect%5B%5D=info_schema.tables&collect%5B%5D=info_schema.tablestats&collect%5B%5D=info_schema.userstats&collect%5B%5D=perf_schema.eventsstatements&collect%5B%5D=perf_schema.file_instances&collect%5B%5D=perf_schema.indexiowaits&collect%5B%5D=perf_schema.tableiowaits": 429; expecting 200; response body: "429 Too Many Requests\n"  agentID=/agent_id/f3d8558b-c6f8-4e15-8d9b-40df654feb71 component=agent-process type=vm_agent
      

      The same issue happens when PMM conflict happens for the same metric but with different resolution.

      As a result affected metrics have gaps.

      Attachments

        Activity

          People

            carlos.salguero Carlos Salguero (Inactive)
            sveta.smirnova Sveta Smirnova
            Votes:
            1 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Smart Checklist