Uploaded image for project: 'Percona Monitoring and Management'
  1. Percona Monitoring and Management
  2. PMM-8421

Listen-port ignored/removed for external services after server update to PMM 2.19 and higher

Details

    • Bug
    • Status: Done
    • High
    • Resolution: Fixed
    • 2.19.0
    • 2.22.0
    • PMM Agent, PMM Server
    • None
    • 4
    • Yes
    • Yes
    • C/S Core

    Description

      After upgrading PMM Server from 2.18 to 2.19 the previously configured ports for all external services is being ignored. The node exporter is working as expected.

      Example log entry:

      Jul 13 10:48:18 envi-test-rmq-1 pmm-agent[1465384]: INFO[2021-07-13T10:48:18.935+02:00] 2021-07-13T08:48:18.935Z error VictoriaMetrics/lib/promscrape/scrapework.go:231 error when scraping “http://127.0.0.1:0/metrics” from job “external-exporter_agent_id_a856cbdb-4a4b-442b-b9af-62957ca2477b_mr-10s” with labels {agent_id="/agent_id/a856cbdb-4a4b-442b-b9af-62957ca2477b",agent_type=“external-exporter”,external_group=“external”,instance="/agent_id/a856cbdb-4a4b-442b-b9af-62957ca2477b",job=“external-exporter_agent_id_a856cbdb-4a4b-442b-b9af-62957ca2477b_mr-10s”,machine_id="/machine_id/a69c5724b306422387bb1b08ee1f2edb",node_id="/node_id/f5dd0e77-405a-4bcf-a3c5-7e82652f954e",node_name=“envi-test-rmq-1”,node_type=“generic”,service_id="/service_id/ec7a45a9-025f-4075-876c-d116c0b6f394",service_name=“envi-test-rmq-1-systemd”,service_type=“external”}: error when scraping “http://127.0.0.1:0/metrics”: dial tcp4 127.0.0.1:0: connect: connection refused; try -enableTCP6 command-line flag if you scrape ipv6 addresses agentID=/agent_id/5c7d46da-b037-4cb9-8e75-02e37de04f1c component=agent-process type=vm_agent

      After removing the service and adding it again, everything seems to work properly again - until pmm-agent is being restarted. Then the same issue occurs.

      This example's external service is being added with

      pmm-admin add external --metrics-path=metrics --service-name=“envi-test-rmq-1-systemd” --listen-port=42009 --scheme=http --metrics-mode=auto

      This issue occurs on different nodes with pmm-client versions up from 2.15 to 2.19, and is still occurring after a client has been updated to 2.19.

      OS: Ubuntu 18.04 LTS and Ubuntu 20.04 LTS.

      No errors have been reported during the server update.

      Edit: After rolling back to PMM Server 2.18 everything is back to normal.

      Edit 2:

      Upgrading to PMM Server 2.20 does not resolve this issue, as well as manually upgrading the docker image.

      A (probably obvious) observation: If the agent's "metrics mode" is "pull", everything works as expected, only "push" agents are affected.

      Another - probably even more obvious - observation: The nodes' vmagent promscrape.config file contains the invalid target information after the server upgrade, e.g. 127.0.0.1:0 instead of 127.0.0.1:42008 - except for the job with the "vmagent" agent type.

      And finally: The "listen_port" in PMM inventory seems to be removed for all/most agents with "push_metrics_enabled: true".

      Attachments

        Issue Links

          Activity

            People

              jiri.ctvrtka Jiří Čtvrtka
              rapli Rainer Plischke
              Votes:
              2 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Smart Checklist