Uploaded image for project: 'Percona Monitoring and Management'
  1. Percona Monitoring and Management
  2. PMM-1760

Very high CPU usage (possible infinite loop) in QAN 1.4.1 w/MongoDB

Details

    Description

      Hey guys,

      I am seeing very high CPU/load average on a host that I recently installed PMM 1.4.1 w/QAN and MongoDB.

      Attached is a screenshot showing percona-qan-agent process taking up basically all of my CPU and causing a load average of almost 2.00 on my 2 x 2.2GHz CPU host. When I stop this process my load average and CPU drops to basically nothing.

      Load Average during the issue I mention: 1.97
      Load Average after stopping QAN: 0.23 (CPU usage disappeared)

      To me this looks like some sort of tight loop. The log file has no warning/errors to suggest what is happening.

      RPM Version

      $ rpm -qai pmm-client
      Name        : pmm-client
      Version     : 1.4.1
      Release     : 1
      Architecture: x86_64
      Install Date: Thu Nov 23 17:18:29 2017
      Group       : Applications/Databases
      Size        : 55411208
      License     : AGPLv3
      Signature   : DSA/SHA1, Mon Oct 30 12:29:28 2017, Key ID 1c4cbdcdcd2efd2a
      Source RPM  : pmm-client-1.4.1-1.src.rpm
      Build Date  : Mon Oct 30 12:29:10 2017
      Build Host  : os-centos-7-x64-1497.ci.percona.com
      Relocations : (not relocatable)
      Vendor      : Percona LLC
      URL         : https://percona.com
      Summary     : Percona Monitoring and Management Client
      Description :
      Percona Monitoring and Management (PMM) is an open-source platform for managing and monitoring MySQL and MongoDB
      performance. It is developed by Percona in collaboration with experts in the field of managed database services,
      support and consulting.
      PMM is a free and open-source solution that you can run in your own environment for maximum security and reliability.
      It provides thorough time-based analysis for MySQL and MongoDB servers to ensure that your data works as efficiently
      as possible.

      config/agent.conf:

      $ cat agent.conf;echo
      {"UUID":"f0685a8d3e8848ac4ae11fa520670177","ApiHostname":"localhost","ApiPath":"/qan-api","ServerUser":"pmm"}

      config/qan-cc1f6bec18a3410e7fde2afe339b2702.conf

      $ cat qan-cc1f6bec18a3410e7fde2afe339b2702.conf;echo
      {
          "UUID": "cc1f6bec18a3410e7fde2afe339b2702",
          "CollectFrom": "",
          "Interval": 60,
          "ExampleQueries": true
      }

      instances/4d2f20aa30c6470f40ddb83ff5c1b099.json

      $ cat 4d2f20aa30c6470f40ddb83ff5c1b099.json ;echo
      {
          "Subsystem": "os",
          "ParentUUID": "",
          "Id": 0,
          "UUID": "4d2f20aa30c6470f40ddb83ff5c1b099",
          "Name": "centos7",
          "DSN": "",
          "Distro": "",
          "Version": "",
          "Created": "2017-11-23T19:14:26Z",
          "Deleted": "0001-01-01T00:00:00Z"
      }

      instances/cc1f6bec18a3410e7fde2afe339b2702.json

      $ cat cc1f6bec18a3410e7fde2afe339b2702.json; echo
      {
          "Subsystem": "mongo",
          "ParentUUID": "4d2f20aa30c6470f40ddb83ff5c1b099",
          "Id": 0,
          "UUID": "cc1f6bec18a3410e7fde2afe339b2702",
          "Name": "centos7",
          "DSN": "mongodb://user:[email protected]:28017",
          "Distro": "MongoDB",
          "Version": "3.4.9-2.9",
          "Created": "2017-11-23T19:14:26Z",
          "Deleted": "1970-01-01T00:00:01Z"
      }

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              tim.vaillancourt Tim Vaillancourt (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 2 days
                  2d

                  Smart Checklist