Uploaded image for project: 'Percona Monitoring and Management'
  1. Percona Monitoring and Management
  2. PMM-7302

The attribute max_alerts is not valid for webhook_configs for AlertManager version included in PMM v2.13.0

Details

    • 1
    • 09 - Server Features
    • Yes
    • Yes
    • Server Features

    Description

      Issue: The attribute max_alerts is not valid for webhook_configs for AlertManager version included in PMM v2.13.0.

      pmm-managed log:

       

      /usr/local/go/src/runtime/asm_amd64.s:1373. ^[[31mcomponent^[[0m=alertmanager
      ^[[36mINFO^[[0m[2021-01-12T10:03:11.846+00:00] STT is not enabled, doing nothing. ^[[36mcomponent^[[0m=checks
      ^[[31mERRO^[[0m[2021-01-12T10:03:12.620+00:00] Checking '/tmp/pmm-managed-config-alertmanager-122710806' FAILED: yaml: unmarshal errors:
       line 14: field max_alerts not found in type config.plain
      amtool: error: failed to validate 1 file(s)
       ^[[31mcomponent^[[0m=alertmanager
      ^[[31mERRO^[[0m[2021-01-12T10:03:12.620+00:00] Failed to update configuration, will retry: exit status 1
      Checking '/tmp/pmm-managed-config-alertmanager-122710806' FAILED: yaml: unmarshal errors:
       line 14: field max_alerts not found in type config.plain
      amtool: error: failed to validate 1 file(s)
      

       

       

      And the attribute send_resolved should be true (default value), not false

      Reproduciable Test case:

      Did test by configuring an external alertmanager for alerting, but it's not needed to reproduce the issue.

       

      • Install PMM-SERVER 2.13.0

       

      # docker create -v /srv --name pmm-data percona/pmm-server:2.13.0 /bin/true
      7b73f35f3ccf2d4297b038a4ae10c88cb7c56e413a8264d30e99eba77eb1a241
      # docker run -d -p 80:80 -p 443:443 --volumes-from pmm-data --name pmm-server --restart always percona/pmm-server:2.13.0
      5f1ed1d08b88d8b79ba2239a27cecea70d89b3eaa98ee43acd7d2ea6fbfd2dcb

       

       

      1. docker exec -it pmm-server /bin/bash
      • Add the following entries to /srv/alertmanager/alertmanager.base.yml 

       

       

      ---
      # You can edit this file; changes will be preserved.
      route:
       receiver: webhook
       routes: []
      receivers:
      - name: 'webhook'
       webhook_configs:
       - url: 'http://127.0.0.1:5001/webhook'
      - name: 'email'
       email_configs:
       - send_resolved: true # Send clear
       to: '[email protected]'
      templates: []
      

       

       

      Restart pmm-server: 

      # docker restart pmm-server
      pmm-server
      # docker exec -it pmm-server supervisorctl status
      alertmanager RUNNING pid 25, uptime 0:00:05
      clickhouse RUNNING pid 14, uptime 0:00:05
      cron RUNNING pid 17, uptime 0:00:05
      dashboard-upgrade EXITED Jan 12 10:02 AM
      dbaas-controller STOPPED Not started
      grafana RUNNING pid 15, uptime 0:00:05
      nginx RUNNING pid 16, uptime 0:00:05
      pmm-agent RUNNING pid 44, uptime 0:00:05
      pmm-managed RUNNING pid 42, uptime 0:00:05
      pmm-update-perform STOPPED Not started
      postgresql RUNNING pid 13, uptime 0:00:05
      prometheus STOPPED Not started
      qan-api2 RUNNING pid 186, uptime 0:00:04
      victoriametrics RUNNING pid 18, uptime 0:00:05
      vmalert RUNNING pid 19, uptime 0:00:05

      check /srv/logs/pmm-managed.log and you see the error for webhook config max_alerts attribute. 

       

      /usr/local/go/src/runtime/asm_amd64.s:1373. ^[[31mcomponent^[[0m=alertmanager
      ^[[36mINFO^[[0m[2021-01-12T10:03:11.846+00:00] STT is not enabled, doing nothing. ^[[36mcomponent^[[0m=checks
      ^[[31mERRO^[[0m[2021-01-12T10:03:12.620+00:00] Checking '/tmp/pmm-managed-config-alertmanager-122710806' FAILED: yaml: unmarshal errors:
       line 14: field max_alerts not found in type config.plain
      amtool: error: failed to validate 1 file(s)
       ^[[31mcomponent^[[0m=alertmanager
      ^[[31mERRO^[[0m[2021-01-12T10:03:12.620+00:00] Failed to update configuration, will retry: exit status 1
      Checking '/tmp/pmm-managed-config-alertmanager-122710806' FAILED: yaml: unmarshal errors:
       line 14: field max_alerts not found in type config.plain
      amtool: error: failed to validate 1 file(s)
      
      github.com/percona/pmm-managed/services/alertmanager.(*Service).validateConfig
       /home/builder/rpm/BUILD/pmm-managed-0db06f75cbb60d688e85a34133f51dd5d140fa25/src/github.com/percona/pmm-managed/services/alertmanager/alertmanager.go:281
      github.com/percona/pmm-managed/services/alertmanager.(*Service).configAndReload
       /home/builder/rpm/BUILD/pmm-managed-0db06f75cbb60d688e85a34133f51dd5d140fa25/src/github.com/percona/pmm-managed/services/alertmanager/alertmanager.go:314
      github.com/percona/pmm-managed/services/alertmanager.(*Service).updateConfiguration
       /home/builder/rpm/BUILD/pmm-managed-0db06f75cbb60d688e85a34133f51dd5d140fa25/src/github.com/percona/pmm-managed/services/alertmanager/alertmanager.go:194
      github.com/percona/pmm-managed/services/alertmanager.(*Service).Run
       /home/builder/rpm/BUILD/pmm-managed-0db06f75cbb60d688e85a34133f51dd5d140fa25/src/github.com/percona/pmm-managed/services/alertmanager/alertmanager.go:162
      main.main.func5
       /home/builder/rpm/BUILD/pmm-managed-0db06f75cbb60d688e85a34133f51dd5d140fa25/src/github.com/percona/pmm-managed/main.go:707
      runtime.goexit
       /usr/local/go/src/runtime/asm_amd64.s:1373. ^[[31mcomponent^[[0m=alertmanager
      

       

      PMM-SERVER LOG: pmm-server_2021-01-12_10-36.zip

      More details from the user in following bug report:

      https://jira.percona.com/browse/PMM-7271?focusedCommentId=275554&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-275554

       

      Actual result:  Error for max_alerts attribute 

      Expected Result: max_alerts attribute should work.

      Attachments

        Issue Links

          Activity

            People

              david.mikus David Mikus (Inactive)
              lalit.choudhary Lalit Choudhary
              Votes:
              2 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Smart Checklist