Uploaded image for project: 'Percona Monitoring and Management'
  1. Percona Monitoring and Management
  2. PMM-5969

Removing Services or Nodes with `pmm-admin ... --force` mode does not stop running agents, VictoriaMetrics continues collecting data from exporters

Details

    Description

      User Impact:

      DOD:

      User impact:

      after removing services , agent continue to consume resources on DB server and victoriametrics still collect data from exporters.
      --------
      Based on code it looks like we don't stop agents when we remove Services or Nodes in force mode using inventory API. Also we don't remove them from prometheus config. 

      Steps to reproduce:

      1. Setup PMM Server and pmm-agent 
      2. Run MySQL Server
      3. Add new service via `pmm-admin add mysql` command
      4. Delete MySQL service via `pmm-admin inventory remove service` with --force flag.
      5. Run `ps -ax | grep mysqld_exporter` and check there is no mysqld_exporter
      6. Check that /etc/victoriametrics-promscrape.yml doesn't have scrape config for deleted exporter

      Actual result;

      After removing service with force flag all agents related to the service are not removed from VictoriaMetrics config.

      After removing node with force flag all agents related to the service are not removed from VictoriaMetrics config.

      Processes related to agents are not stopped after we remove them from DB. 

      Expected result:

      After removing service with force flag all agents related to the service should be removed from VictoriaMetrics config.

      After removing node with force flag all agents related to the service should be removed from VictoriaMetrics config.

      Processes related to agents are stopped after we remove them from DB.

      Workaround:

      Send empty request to UpdateSettings API. 

      Details:

       

      Attachments

        Issue Links

          Activity

            People

              maksym.hilliaka Maksym Hilliaka (Inactive)
              nurlan.moldomurov Nurlan Moldomurov
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Smart Checklist