Uploaded image for project: 'Percona Monitoring and Management'
  1. Percona Monitoring and Management
  2. PMM-1760

Very high CPU usage (possible infinite loop) in QAN 1.4.1 w/MongoDB

    Details

      Description

      Hey guys,

      I am seeing very high CPU/load average on a host that I recently installed PMM 1.4.1 w/QAN and MongoDB.

      Attached is a screenshot showing percona-qan-agent process taking up basically all of my CPU and causing a load average of almost 2.00 on my 2 x 2.2GHz CPU host. When I stop this process my load average and CPU drops to basically nothing.

      Load Average during the issue I mention: 1.97
      Load Average after stopping QAN: 0.23 (CPU usage disappeared)

      To me this looks like some sort of tight loop. The log file has no warning/errors to suggest what is happening.

      RPM Version

      $ rpm -qai pmm-client
      Name        : pmm-client
      Version     : 1.4.1
      Release     : 1
      Architecture: x86_64
      Install Date: Thu Nov 23 17:18:29 2017
      Group       : Applications/Databases
      Size        : 55411208
      License     : AGPLv3
      Signature   : DSA/SHA1, Mon Oct 30 12:29:28 2017, Key ID 1c4cbdcdcd2efd2a
      Source RPM  : pmm-client-1.4.1-1.src.rpm
      Build Date  : Mon Oct 30 12:29:10 2017
      Build Host  : os-centos-7-x64-1497.ci.percona.com
      Relocations : (not relocatable)
      Vendor      : Percona LLC
      URL         : https://percona.com
      Summary     : Percona Monitoring and Management Client
      Description :
      Percona Monitoring and Management (PMM) is an open-source platform for managing and monitoring MySQL and MongoDB
      performance. It is developed by Percona in collaboration with experts in the field of managed database services,
      support and consulting.
      PMM is a free and open-source solution that you can run in your own environment for maximum security and reliability.
      It provides thorough time-based analysis for MySQL and MongoDB servers to ensure that your data works as efficiently
      as possible.

      config/agent.conf:

      $ cat agent.conf;echo
      {"UUID":"f0685a8d3e8848ac4ae11fa520670177","ApiHostname":"localhost","ApiPath":"/qan-api","ServerUser":"pmm"}

      config/qan-cc1f6bec18a3410e7fde2afe339b2702.conf

      $ cat qan-cc1f6bec18a3410e7fde2afe339b2702.conf;echo
      {
          "UUID": "cc1f6bec18a3410e7fde2afe339b2702",
          "CollectFrom": "",
          "Interval": 60,
          "ExampleQueries": true
      }

      instances/4d2f20aa30c6470f40ddb83ff5c1b099.json

      $ cat 4d2f20aa30c6470f40ddb83ff5c1b099.json ;echo
      {
          "Subsystem": "os",
          "ParentUUID": "",
          "Id": 0,
          "UUID": "4d2f20aa30c6470f40ddb83ff5c1b099",
          "Name": "centos7",
          "DSN": "",
          "Distro": "",
          "Version": "",
          "Created": "2017-11-23T19:14:26Z",
          "Deleted": "0001-01-01T00:00:00Z"
      }

      instances/cc1f6bec18a3410e7fde2afe339b2702.json

      $ cat cc1f6bec18a3410e7fde2afe339b2702.json; echo
      {
          "Subsystem": "mongo",
          "ParentUUID": "4d2f20aa30c6470f40ddb83ff5c1b099",
          "Id": 0,
          "UUID": "cc1f6bec18a3410e7fde2afe339b2702",
          "Name": "centos7",
          "DSN": "mongodb://user:pwd@localhost:28017",
          "Distro": "MongoDB",
          "Version": "3.4.9-2.9",
          "Created": "2017-11-23T19:14:26Z",
          "Deleted": "1970-01-01T00:00:01Z"
      }

        Smart Checklist

          Attachments

            Issue Links

              Activity

                People

                • Assignee:
                  Unassigned
                  Reporter:
                  tim.vaillancourt Tim Vaillancourt (Inactive)
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  5 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Time Tracking

                    Estimated:
                    Original Estimate - Not Specified
                    Not Specified
                    Remaining:
                    Remaining Estimate - 0 minutes
                    0m
                    Logged:
                    Time Spent - 2 days
                    2d