Resolution: Cannot Reproduce
Affects Version/s: 2.1.0
Fix Version/s: None
Running on Kubernetes: GKE 1.14.9
OS: Google Cloud COS
Bash script that starts the pmm2 agent as each MySQL instance comes upRunning on Kubernetes: GKE 1.14.9 OS: Google Cloud COS Bash script that starts the pmm2 agent as each MySQL instance comes up set -ex # Redirect pmm-client data to persistent volume if [ ! -d /vtdataroot/pmm2 ]; then FIRST_RUN=1 mkdir -p /vtdataroot/pmm2 fi pmm-agent setup --config-file=/vtdataroot/pmm2/pmm-agent.yaml --server-address=pmm2.vitess.svc.cluster.local --server-username=admin --server-password=admin --server-insecure-tls --force "$(hostname).vitess" container "$(hostname)" # wait for mysql to be available before starting the agent until [ -e /vtdataroot/tabletdata/mysql.sock ]; do echo "Waiting for mysql.sock file" sleep 1 done # start the agent in the background pmm-agent --config-file=/vtdataroot/pmm2/pmm-agent.yaml & # give the agent time to start sleep 5 pmm-admin add mysql --query-source=slowlog --username=nz_monitoring "$(hostname)" 127.0.0.1:3306 # keep the container alive but still responsive to stop requests trap : TERM INT; sleep infinity & wait
I have 76 nodes with agents reporting to PMM 2.1.0. There are agent ids for:
- PMM Agent
- Node exporter
- MySQL exporter
- Qan MySQL Slowlog Agent
The clients seem to be working and do not report any errors. There do not seem to be any obvious server side errors either, however when I go to Grafana, it only shows 1 system being monitored - the PMM Server itself. The agents are all listed in the PMM Inventory page.
When I first attached clients, I saw 5-6 actually register and show metrics in pmm2, but after restarting them, eventually they all disappeared.
Though I'm not able to see metrics in Prometheus, I am getting QAN slow log analytics from all the nodes, so at least some data is making it through.