Uploaded image for project: 'Percona Monitoring and Management'
  1. Percona Monitoring and Management
  2. PMM-7326

Pmm-admin logs too verbose into syslog Percona Monitoring and Management (PMM) PMM 2.x

Details

    Description

      User story:
      As a PMM user, I need to be explicitly set a more detailed level of logging if this is needed so that by default people will have fewer logs and this will simplify troubleshooting for them and let them store more meaningful logs

      UI/UX:
      > pmm-admin config ... --log-level=INFO

       

      while the default should be `–log–level=WARN`, which will also log all errors.

      Acceptance criteria

      • User able to pass an additional flag to log INFO messages
      • By default, INFO is not logged
      • --debug flag is working but,  will be deprecated in favor of --log-level flag
      • log-level passed by pmm-admin inherited by all exporters and components 
      • possible values are:  [debug, info, warn, error, fatal] 
      • Documentation:
        • Logs levels explained on the "configuration stage" and we recommend more detailed levels for dev env.
        • We have a clear warning on documentation about log rotation on OS and provide some links to articles for major OS
        • we have links to official documentation about changing logs location for major supported OS

      Out of scope:

      • Logrotate implementation (we expect this will be done on OS level)

      Suggested implementation:
      TBD

      How to test:
      Case 1:
      Run pmm-managed with MongoDB, MySQL, PostgreSQL and ProxySQL

      Run pmm-agent in side terminal (1)
      Run pmm-admin in side terminal (2) with:

      • pmm-admin add mongodb --credentials --log-level=info
      • pmm-admin add mysql --credentials --log-level=info
      • pmm-admin add postgresql --credentials --log-level=info
      • pmm-admin add proxysql --credentials ---log-level=info

      Check:
      In Grafana in "inventory" tab check that log-level of added exporters are correct
      In side terminal (1) check that logs from exporters has correct log-level >=(info, warn, error, fatal)

      Case 2:
      Same, but run pmm-admin in side terminal (2) with:

      • pmm-admin add mongodb --credentials {no log level to use default}
      • pmm-admin add mysql --credentials --log-level=warn
      • pmm-admin add postgresql --credentials --log-level=error
      • pmm-admin add proxysql --credentials ---log-level=fatal

      Check: same

      Instead of "--credentials" use real credentials to add exporter

      Original report:
       

      Hi,

      we are using PMM 2.13 version. 

      Please see below for more details

      https://forums.percona.com/t/pmm-admin-logs-too-verbose-into-syslog/8384/7

       

      The issue is with version of systemd 219 and also with centos 7. we are unable to choose diffrent log file location for pmm logs. it is using syslog where things are too verbose and chances of missing critical system messages are missed. Please let us know if there is any other way of putting PMM logs on diffrent location.

       

      Here is what  i have . for the system

       
      [Unit]
      Description=pmm-agent
      After=time-sync.target network.target

      [Service]
      Type=simple
      ExecStart=/usr/sbin/pmm-agent --config-file=/usr/local/percona/pmm2/config/pmm-agent.yaml
      Restart=always
      RestartSec=2s
      StandardError=file:/var/log/pmm-agent.log
      [Install]
      WantedBy=multi-user.target

      [[email protected] log]# touch /var/log/pmm-agent.log
      [[email protected] log]# systemctl daemon-reload
      [[email protected] log]# systemctl restart pmm-agent
      [[email protected] log]# journalctl -f
      – Logs begin at Thu 2020-10-15 23:14:58 CDT. –
      Jan 08 11:32:20<hostname> pmm-agent[28906]: INFO[2021-01-08T11:32:20.653-06:00] 2021-01-08T17:32:20.653Z info VictoriaMetrics/app/vmagent/main.go:111 started vmagent in 0.013 seconds agentID=/agent_id/d89645e9-e902-4995-b4b4-2dfd176f7303 component=agent-process type=vm_agent
      Jan 08 11:32:20 dbsl01.q2dc.local pmm-agent[28906]

       

       

      Update
      som our exporters have: 
      --log.level="info" Only log messages with the given severity or above. Valid levels: [debug, info, warn,
      error, fatal] 

      Attachments

        1. Grafana-home-PMM-2.28.0.png
          Grafana-home-PMM-2.28.0.png
          21 kB
        2. image-2022-05-11-13-57-25-373.png
          image-2022-05-11-13-57-25-373.png
          37 kB
        3. image-2022-05-19-02-38-43-049.png
          image-2022-05-19-02-38-43-049.png
          42 kB
        4. image-2022-05-19-02-39-00-923.png
          image-2022-05-19-02-39-00-923.png
          113 kB
        5. image-2022-05-19-02-39-13-142.png
          image-2022-05-19-02-39-13-142.png
          33 kB
        6. image-2022-05-19-03-02-27-318.png
          image-2022-05-19-03-02-27-318.png
          153 kB
        7. image-2022-05-19-03-02-41-721.png
          image-2022-05-19-03-02-41-721.png
          46 kB
        8. image-2022-05-19-03-16-34-666.png
          image-2022-05-19-03-16-34-666.png
          50 kB
        9. image-2022-05-19-03-18-05-229.png
          image-2022-05-19-03-18-05-229.png
          135 kB
        10. image-2022-05-19-03-31-01-475.png
          image-2022-05-19-03-31-01-475.png
          34 kB
        11. image-2022-05-19-03-31-22-213.png
          image-2022-05-19-03-31-22-213.png
          137 kB
        12. image-2022-05-19-03-40-30-795.png
          image-2022-05-19-03-40-30-795.png
          34 kB
        13. image-2022-05-19-03-40-54-266.png
          image-2022-05-19-03-40-54-266.png
          135 kB
        14. image-2022-05-19-03-48-54-665.png
          image-2022-05-19-03-48-54-665.png
          33 kB
        15. image-2022-05-19-03-49-13-640.png
          image-2022-05-19-03-49-13-640.png
          133 kB
        16. image-2022-05-19-03-56-20-904.png
          image-2022-05-19-03-56-20-904.png
          33 kB
        17. image-2022-05-19-03-56-42-328.png
          image-2022-05-19-03-56-42-328.png
          135 kB
        18. image-2022-05-19-04-01-11-320.png
          image-2022-05-19-04-01-11-320.png
          135 kB
        19. image-2022-05-23-02-44-32-835.png
          image-2022-05-23-02-44-32-835.png
          134 kB
        20. image-2022-05-23-02-46-10-985.png
          image-2022-05-23-02-46-10-985.png
          63 kB
        21. image-2022-05-23-02-46-30-909.png
          image-2022-05-23-02-46-30-909.png
          23 kB
        22. image-2022-05-23-02-51-30-043.png
          image-2022-05-23-02-51-30-043.png
          10 kB
        23. image-2022-05-23-03-12-26-192.png
          image-2022-05-23-03-12-26-192.png
          29 kB
        24. image-2022-05-23-03-12-43-050.png
          image-2022-05-23-03-12-43-050.png
          115 kB
        25. image-2022-05-23-03-18-25-410.png
          image-2022-05-23-03-18-25-410.png
          62 kB
        26. image-2022-05-23-03-18-59-119.png
          image-2022-05-23-03-18-59-119.png
          75 kB
        27. image-2022-05-23-03-19-12-678.png
          image-2022-05-23-03-19-12-678.png
          27 kB
        28. testing-upgrade.png
          testing-upgrade.png
          17 kB
        29. upgrade-2-mysql-exporter.png
          upgrade-2-mysql-exporter.png
          86 kB
        30. upgrade-2-mysql-exporter-after.png
          upgrade-2-mysql-exporter-after.png
          60 kB
        31. upgrade-2-postgres.png
          upgrade-2-postgres.png
          51 kB
        32. upgrade-2-postgres-exporter-after.png
          upgrade-2-postgres-exporter-after.png
          50 kB

        Issue Links

          Activity

            People

              Unassigned Unassigned
              dacharya Dipesh Acharya
              Votes:
              10 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Smart Checklist