Uploaded image for project: 'Percona Monitoring and Management'
  1. Percona Monitoring and Management
  2. PMM-4557

Limit mysqld_exporter to at most 1 set of pending queries running per Prometheus sample interval

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Confirmation
    • Priority: High
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: MySQLd_Exporter
    • Labels:
    • Story Points:
      2

      Description

      We've had a problem in production where PMM contributes to database overload. Here are the steps for how that happened:

      1. Our production database got very busy doing a massive query requiring file sort to a temporary table. This saturated the IOPS on the drive (EBS in AWS).
      2. Prometheus hit mysqld_exporter for a sample interval (once a minute I believe).
      3. The mysqld_exporter queries took more than a minute to run.
      4. Go to 2.

      The result was an unbounded stack of mysqld_exporter queries all running at the same time the database was trying to finish the query from (1).

      It looks from source like mysqld_exporter tries to protect against this with the `X-Prometheus-Scrape-Timeout-Seconds` timeout. But I didn't see that header being passed. And in any case, that is independent. If that header isn't passed, or that protect doesn't work, the protection in the PR will keep the infinite stack from accumulating.

        Smart Checklist

          Attachments

            Issue Links

              Activity

                People

                Assignee:
                nurlan.moldomurov Nurlan Moldomurov
                Reporter:
                colindkelley@gmail.com Colin D Kelley
                Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                  Dates

                  Created:
                  Updated: