Uploaded image for project: 'Percona Monitoring and Management'
  1. Percona Monitoring and Management
  2. PMM-9503

Remove prometheus from supervisord config

Details

    • Admin & Maintenance Task
    • Status: Pending Release
    • Medium
    • Resolution: Done
    • 2.26.0
    • 2.31.0
    • PMM ManageD
    • 1
    • Yes
    • Yes
    • Architecture & Foundation

    Description

      Problem:

      After we migrated from Prometheus to VictoriaMetrics, we never attempted to remove prometheus.ini config for supervisor. If the User sequentially runs `supervisorctl stop all` and `supervisorctl start all`, they will notice that supervisor tried and failed to start Prometheus.

      % supervisorctl start all
      ...
      prometheus: ERROR (spawn error)

      Later, when they try to print supervisord's status, they will see that supervisor has been relentlessly trying to launch prometheus' daemon, but all attempts failed, since on PMM versions starting from 2.12.0 `prometheus` is not installed.

      % supervisorctl status
      ...
      prometheus                       BACKOFF   Exited too quickly (process log may have details)
      
      # a bit later
      % supervisorctl status
      prometheus                       FATAL     Exited too quickly (process log may have details)

      This is definitely redundant and the User would benefit if we remove this noice from the system.

       

      Solution:

      Remove prometheus.ini it from PMM.

       

      Suggested implementation:

      • Leverage ansible to remove the package named percona-prometheus, which may have been kept on versions prior to 2.12.0
      • Remove the generation of `prometheus.ini` config from `pmm-managed`, which is responsible for it now.

       

      How to test:

      • after executing `docker exec -it pmm-server sh -c "supervisorctl status"` against a running pmm-server, you should get:
      % docker exec -it pmm-server sh -c "supervisorctl status"
      alertmanager                     RUNNING   pid 280, uptime 0:00:29
      clickhouse                       RUNNING   pid 64, uptime 0:00:36
      dbaas-controller                 RUNNING   pid 326, uptime 0:00:28
      grafana                          RUNNING   pid 68, uptime 0:00:36
      nginx                            RUNNING   pid 69, uptime 0:00:35
      pmm-agent                        RUNNING   pid 101, uptime 0:00:35
      pmm-managed                      RUNNING   pid 92, uptime 0:00:35
      pmm-update-perform               STOPPED   Not started
      pmm-update-perform-init          EXITED    Aug 06 01:19 AM
      postgresql                       RUNNING   pid 63, uptime 0:00:36
      qan-api2                         RUNNING   pid 237, uptime 0:00:32
      victoriametrics                  RUNNING   pid 73, uptime 0:00:35
      vmalert                          RUNNING   pid 75, uptime 0:00:35 
      • be sure that `prometheus` is not listed
      • be sure that `prometheus.ini` is absent in `/etc/supervisord.d/`

      Attachments

        Issue Links

          Activity

            People

              alexander.tymchuk Alexander Tymchuk
              alexander.tymchuk Alexander Tymchuk
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Smart Checklist