Uploaded image for project: 'Percona Monitoring and Management'
  1. Percona Monitoring and Management
  2. PMM-10175

pmm-agent: missing logs

    XMLWordPrintable

Details

    • Yes
    • 1
    • Yes
    • Yes
    • No
    • [obsolete] Implex

    Description

      User impact:

      User cannot see exporter logs even with lowest log level

      Steps to reproduce:

      Start PMM dev-latest
      Add PS to monitoring with info log level
      Add PS to monitoring with incorrect password + skip connection check + warning log level

      pmm-admin add mysql --username=root --password=wrong --skip-connection-check --query-source=perfschema wrongpass-default-warn --log-level=warn 127.0.0.1:43306

      Actual result:

      No mysql_exporter logs at all, only from qan_mysql_perfschema_agent:

      [ec2-user@ip-10-178-1-55 ~]$ journalctl -u pmm-agent -f
-- Logs begin at Tue 2022-06-07 08:32:39 UTC. --
Jun 14 09:44:51 ip-10-178-1-55.us-east-2.compute.internal pmm-agent[11435]: ERRO[2022-06-14T09:44:51.647+00:00] failed to query events_statements_history: Error 1045: Access denied for user 'root'@'172.17.0.1' (using password: YES)  agentID=/agent_id/ef7ebcec-68d0-4a22-864d-1ec9dcf45e92 component=agent-builtin type=qan_mysql_perfschema_agent
Jun 14 09:44:53 ip-10-178-1-55.us-east-2.compute.internal pmm-agent[11435]: ERRO[2022-06-14T09:44:53.308+00:00] failed to query events_statements_history: Error 1045: Access denied for user 'root'@'172.17.0.1' (using password: YES)  agentID=/agent_id/b221ae35-7080-4534-9206-7a83a5c19a76 component=agent-builtin type=qan_mysql_perfschema_agent
Jun 14 09:44:56 ip-10-178-1-55.us-east-2.compute.internal pmm-agent[11435]: ERRO[2022-06-14T09:44:56.647+00:00] failed to query events_statements_history: Error 1045: Access denied for user 'root'@'172.17.0.1' (using password: YES)  agentID=/agent_id/ef7ebcec-68d0-4a22-864d-1ec9dcf45e92 component=agent-builtin type=qan_mysql_perfschema_agent
Jun 14 09:44:58 ip-10-178-1-55.us-east-2.compute.internal pmm-agent[11435]: ERRO[2022-06-14T09:44:58.307+00:00] failed to query events_statements_history: Error 1045: Access denied for user 'root'@'172.17.0.1' (using password: YES)  agentID=/agent_id/b221ae35-7080-4534-9206-7a83a5c19a76 component=agent-builtin type=qan_mysql_perfschema_agent
Jun 14 09:45:00 ip-10-178-1-55.us-east-2.compute.internal pmm-agent[11435]: ERRO[2022-06-14T09:45:00.012+00:00] failed to query events_statements_summary_by_digest: Error 1045: Access denied for user 'root'@'172.17.0.1' (using password: YES)  agentID=/agent_id/ef7ebcec-68d0-4a22-864d-1ec9dcf45e92 component=agent-builtin type=qan_mysql_perfschema_agent
Jun 14 09:45:00 ip-10-178-1-55.us-east-2.compute.internal pmm-agent[11435]: ERRO[2022-06-14T09:45:00.034+00:00] failed to query events_statements_summary_by_digest: Error 1045: Access denied for user 'root'@'172.17.0.1' (using password: YES)  agentID=/agent_id/b221ae35-7080-4534-9206-7a83a5c19a76 component=agent-builtin type=qan_mysql_perfschema_agent
Jun 14 09:45:01 ip-10-178-1-55.us-east-2.compute.internal pmm-agent[11435]: ERRO[2022-06-14T09:45:01.645+00:00] failed to query events_statements_history: Error 1045: Access denied for user 'root'@'172.17.0.1' (using password: YES)  agentID=/agent_id/ef7ebcec-68d0-4a22-864d-1ec9dcf45e92 component=agent-builtin type=qan_mysql_perfschema_agent 

[ec2-user@ip-10-178-1-55 ~]$ pmm-admin list
Service type        Service name                         Address and port        Service ID
MySQL               correct-password-default-info        127.0.0.1:43306         /service_id/3e48360c-3b3c-463e-a73c-b56e69f4be33
MySQL               correct-password-default-warn        127.0.0.1:43306         /service_id/8f668eab-cd4a-469e-9733-ba8ba7e2f697
MySQL               wrongpass-info                       127.0.0.1:43306         /service_id/ddc02611-122e-4164-8f09-062be67b230d
MySQL               wrongpass-default-warn               127.0.0.1:43306         /service_id/f64269d3-31a1-4547-beb0-d187377bb234Agent type                    Status           Metrics Mode        Agent ID                                              Service ID                                              Port
pmm_agent                     Connected                            /agent_id/6b8c93b0-4747-407c-a51a-0777c26f28ed                                                                0
node_exporter                 Running          push                /agent_id/9e12f87a-2cd5-44c8-bc8d-9e48f63cc158                                                                42001
mysqld_exporter               Running          push                /agent_id/06dec311-166a-455d-a0f6-497ec4258852        /service_id/ddc02611-122e-4164-8f09-062be67b230d        42007
mysqld_exporter               Running          push                /agent_id/b04821ff-d065-4d65-a8a7-3bd685992a23        /service_id/3e48360c-3b3c-463e-a73c-b56e69f4be33        42009
mysqld_exporter               Running          push                /agent_id/b24807b1-1c84-45ae-af9d-60714a498c54        /service_id/f64269d3-31a1-4547-beb0-d187377bb234        42006
mysqld_exporter               Running          push                /agent_id/cc6a54ec-8015-45ec-9060-fa7156e2b077        /service_id/8f668eab-cd4a-469e-9733-ba8ba7e2f697        42008
mysql_perfschema_agent        Running                              /agent_id/19a7d03f-688b-4b6e-873d-87259ae2db1c        /service_id/3e48360c-3b3c-463e-a73c-b56e69f4be33        0
mysql_perfschema_agent        Waiting                              /agent_id/b221ae35-7080-4534-9206-7a83a5c19a76        /service_id/ddc02611-122e-4164-8f09-062be67b230d        0
mysql_perfschema_agent        Waiting                              /agent_id/ef7ebcec-68d0-4a22-864d-1ec9dcf45e92        /service_id/f64269d3-31a1-4547-beb0-d187377bb234        0
mysql_perfschema_agent        Running                              /agent_id/fb2ea795-97c5-4b75-ab3f-526e80945335        /service_id/8f668eab-cd4a-469e-9733-ba8ba7e2f697        0
vmagent                       Running          push                /agent_id/5c5744b1-1558-411d-aa0c-613a66157c17                                                                42000[ec2-user@ip-10-178-1-55 ~]$
[ec2-user@ip-10-178-1-55 ~]$
[ec2-user@ip-10-178-1-55 ~]$ journalctl -u pmm-agent | grep /agent_id/b04821ff-d065-4d65-a8a7-3bd685992a23
[ec2-user@ip-10-178-1-55 ~]$
[ec2-user@ip-10-178-1-55 ~]$ journalctl -u pmm-agent | grep /agent_id/06dec311-166a-455d-a0f6-497ec4258852
[ec2-user@ip-10-178-1-55 ~]$
[ec2-user@ip-10-178-1-55 ~]$ journalctl -u pmm-agent | grep /agent_id/b24807b1-1c84-45ae-af9d-60714a498c54
[ec2-user@ip-10-178-1-55 ~]$

      Expected result:

      With info level and correct password you should be able to see info messages like these:

      Jun 14 09:17:15 ip-10-178-0-122.us-east-2.compute.internal pmm-agent[10914]: INFO[2022-06-14T09:17:15.057+00:00] time="2022-06-14T09:17:15Z" level=info msg="database version 8.0.28-19, distro Percona Server (GPL), Release 19, Revision 31e88966cd3" source="slave_status.go:72"  agentID=/agent_id/b64e252f-f7a9-4b1f-b19e-ead295681df2 component=agent-process type=mysqld_exporter

      And also errors due to incorrect password:

      Jun 14 09:17:16 ip-10-178-0-122.us-east-2.compute.internal pmm-agent[10914]: INFO[2022-06-14T09:17:16.301+00:00] time="2022-06-14T09:17:16Z" level=error msg="Error pinging mysqld: Error 1045: Access denied for user 'root'@'172.17.0.1' (using password: YES)" source="exporter.go:88"  agentID=/agent_id/209c1faa-3c93-4849-80f1-f12a10716e2a component=agent-process type=mysqld_exporter

      Workaround:

       

      Details:

      Probably caused by https://jira.percona.com/browse/PMM-7326 

      Attachments

        Issue Links

          blocks

          Bug - A problem which impairs or prevents the functions of the product. PMM-8622 mysqld_exporter's chooseQuery too verbose

          • Low - Minor issues that have minimal impact on the system's functionality or usability.
          • Done

          Activity

            People

              Unassigned Unassigned
              beata.handzelova1 Beata Handzelova (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Smart Checklist