Uploaded image for project: 'Percona Monitoring and Management'
  1. Percona Monitoring and Management
  2. PMM-8421

Listen-port ignored/removed for external services after server update to PMM 2.19 and higher


    • Bug
    • Status: Done
    • High
    • Resolution: Fixed
    • 2.19.0
    • 2.22.0
    • PMM Agent, PMM Server
    • None
    • 4
    • Yes
    • Yes
    • C/S Core


      After upgrading PMM Server from 2.18 to 2.19 the previously configured ports for all external services is being ignored. The node exporter is working as expected.

      Example log entry:

      Jul 13 10:48:18 envi-test-rmq-1 pmm-agent[1465384]: INFO[2021-07-13T10:48:18.935+02:00] 2021-07-13T08:48:18.935Z error VictoriaMetrics/lib/promscrape/scrapework.go:231 error when scraping “” from job “external-exporter_agent_id_a856cbdb-4a4b-442b-b9af-62957ca2477b_mr-10s” with labels {agent_id="/agent_id/a856cbdb-4a4b-442b-b9af-62957ca2477b",agent_type=“external-exporter”,external_group=“external”,instance="/agent_id/a856cbdb-4a4b-442b-b9af-62957ca2477b",job=“external-exporter_agent_id_a856cbdb-4a4b-442b-b9af-62957ca2477b_mr-10s”,machine_id="/machine_id/a69c5724b306422387bb1b08ee1f2edb",node_id="/node_id/f5dd0e77-405a-4bcf-a3c5-7e82652f954e",node_name=“envi-test-rmq-1”,node_type=“generic”,service_id="/service_id/ec7a45a9-025f-4075-876c-d116c0b6f394",service_name=“envi-test-rmq-1-systemd”,service_type=“external”}: error when scraping “”: dial tcp4 connect: connection refused; try -enableTCP6 command-line flag if you scrape ipv6 addresses agentID=/agent_id/5c7d46da-b037-4cb9-8e75-02e37de04f1c component=agent-process type=vm_agent

      After removing the service and adding it again, everything seems to work properly again - until pmm-agent is being restarted. Then the same issue occurs.

      This example's external service is being added with

      pmm-admin add external --metrics-path=metrics --service-name=“envi-test-rmq-1-systemd” --listen-port=42009 --scheme=http --metrics-mode=auto

      This issue occurs on different nodes with pmm-client versions up from 2.15 to 2.19, and is still occurring after a client has been updated to 2.19.

      OS: Ubuntu 18.04 LTS and Ubuntu 20.04 LTS.

      No errors have been reported during the server update.

      Edit: After rolling back to PMM Server 2.18 everything is back to normal.

      Edit 2:

      Upgrading to PMM Server 2.20 does not resolve this issue, as well as manually upgrading the docker image.

      A (probably obvious) observation: If the agent's "metrics mode" is "pull", everything works as expected, only "push" agents are affected.

      Another - probably even more obvious - observation: The nodes' vmagent promscrape.config file contains the invalid target information after the server upgrade, e.g. instead of - except for the job with the "vmagent" agent type.

      And finally: The "listen_port" in PMM inventory seems to be removed for all/most agents with "push_metrics_enabled: true".


        Issue Links



              jiri.ctvrtka Jiří Čtvrtka
              rapli Rainer Plischke
              2 Vote for this issue
              8 Start watching this issue



                Smart Checklist