After upgrading PMM Server from 2.18 to 2.19 the previously configured ports for all external services is being ignored. The node exporter is working as expected.
Example log entry:
After removing the service and adding it again, everything seems to work properly again - until pmm-agent is being restarted. Then the same issue occurs.
This example's external service is being added with
This issue occurs on different nodes with pmm-client versions up from 2.15 to 2.19, and is still occurring after a client has been updated to 2.19.
OS: Ubuntu 18.04 LTS and Ubuntu 20.04 LTS.
No errors have been reported during the server update.
Edit: After rolling back to PMM Server 2.18 everything is back to normal.
Upgrading to PMM Server 2.20 does not resolve this issue, as well as manually upgrading the docker image.
A (probably obvious) observation: If the agent's "metrics mode" is "pull", everything works as expected, only "push" agents are affected.
Another - probably even more obvious - observation: The nodes' vmagent promscrape.config file contains the invalid target information after the server upgrade, e.g. 127.0.0.1:0 instead of 127.0.0.1:42008 - except for the job with the "vmagent" agent type.
And finally: The "listen_port" in PMM inventory seems to be removed for all/most agents with "push_metrics_enabled: true".