Uploaded image for project: 'Percona Monitoring and Management'
  1. Percona Monitoring and Management
  2. PMM-2100

rds_exporter Crashed after Running for Several Minutes

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Priority: High
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.8.0
    • Component/s: None
    • Labels:
      None

      Description

      RDS exporter become crashed after several minutes.
      version - current master a9b0970

      STR:
      1. create config file
      2. run command

      # cat /etc/percona-rds-exporter.yml
      ---
      instances:
      - region: us-east-1
        instance: rds-aurora1
        type: mysql
        aws_access_key: KEY
        aws_secret_key: SECRET
      - region: us-east-1
        instance: rds-mysql57
        type: mysql
        aws_access_key: KEY
        aws_secret_key: SECRET
      
      # /usr/sbin/rds_exporter --config.file=/etc/percona-rds-exporter.yml
      

      expected result: no crashes

      current result: rds_exporter raise panic after several minutes

      time="2018-02-16T12:20:24Z" level=info msg="Starting RDS exporter (version=, branch=, revision=)" source="main.go:42"
      time="2018-02-16T12:20:24Z" level=info msg="Build context (go=go1.9.4, user=, date=)" source="main.go:43"
      time="2018-02-16T12:20:24Z" level=info msg="RDS exporter started" source="main.go:90"
      time="2018-02-16T12:20:34Z" level=error msg="unknown metric rdsosmetrics_swap_out" source="enhanced.go:240"
      time="2018-02-16T12:20:34Z" level=error msg="unknown metric rdsosmetrics_swap_in" source="enhanced.go:240"
      time="2018-02-16T12:20:34Z" level=error msg="unknown metric rdsosmetrics_swap_out" source="enhanced.go:240"
      time="2018-02-16T12:20:34Z" level=error msg="unknown metric rdsosmetrics_swap_in" source="enhanced.go:240"
      time="2018-02-16T12:20:37Z" level=error msg="no events in region us-east-1 for instance rds-aurora1" source="enhanced.go:108"
      time="2018-02-16T12:20:37Z" level=error msg="no events in region us-east-1 for instance rds-aurora1" source="enhanced.go:108"
      time="2018-02-16T12:20:42Z" level=error msg="unknown metric rdsosmetrics_swap_out" source="enhanced.go:240"
      time="2018-02-16T12:20:42Z" level=error msg="unknown metric rdsosmetrics_swap_in" source="enhanced.go:240"
      time="2018-02-16T12:20:42Z" level=error msg="unknown metric rdsosmetrics_swap_in" source="enhanced.go:240"
      time="2018-02-16T12:20:42Z" level=error msg="unknown metric rdsosmetrics_swap_out" source="enhanced.go:240"
      time="2018-02-16T12:20:44Z" level=error msg="no events in region us-east-1 for instance rds-aurora1" source="enhanced.go:108"
      time="2018-02-16T12:20:45Z" level=error msg="no events in region us-east-1 for instance rds-aurora1" source="enhanced.go:108"
      time="2018-02-16T12:20:51Z" level=error msg="unknown metric rdsosmetrics_swap_out" source="enhanced.go:240"
      time="2018-02-16T12:20:51Z" level=error msg="unknown metric rdsosmetrics_swap_in" source="enhanced.go:240"
      panic: inconsistent label cardinality
      
      goroutine 199 [running]:
      github.com/percona/rds_exporter/vendor/github.com/prometheus/client_golang/prometheus.MustNewConstMetric(0xc4201915e0, 0x2, 0x418d74dd80000000, 0xc4200c6460, 0x2, 0x2, 0xc4203bb960, 0x14)
      	/builddir/build/BUILD/rds_exporter-a9b0970f99f579ca25b0fa763e413ed8d5ca3b52/src/github.com/percona/rds_exporter/vendor/github.com/prometheus/client_golang/prometheus/value.go:172 +0xb3
      github.com/percona/rds_exporter/enhanced.(*Exporter).sendMetric(0xc42004d200, 0xc42033e7e0, 0xc420017d90, 0xb, 0xc420017d70, 0x9, 0xc420019480, 0x14, 0xc420070540, 0x28, ...)
      	/builddir/build/BUILD/rds_exporter-a9b0970f99f579ca25b0fa763e413ed8d5ca3b52/src/github.com/percona/rds_exporter/enhanced/enhanced.go:249 +0x244
      github.com/percona/rds_exporter/enhanced.(*Exporter).collectMapValue(0xc42004d200, 0xc42033e7e0, 0xc420017d90, 0xb, 0xc420017d70, 0x9, 0xc420019480, 0x14, 0xc420070540, 0x28, ...)
      	/builddir/build/BUILD/rds_exporter-a9b0970f99f579ca25b0fa763e413ed8d5ca3b52/src/github.com/percona/rds_exporter/enhanced/enhanced.go:230 +0x1a4
      github.com/percona/rds_exporter/enhanced.(*Exporter).collectValue(0xc42004d200, 0xc42033e7e0, 0xc420017d90, 0xb, 0xc420017d70, 0x9, 0xc420019480, 0x14, 0xc420070540, 0x28, ...)
      	/builddir/build/BUILD/rds_exporter-a9b0970f99f579ca25b0fa763e413ed8d5ca3b52/src/github.com/percona/rds_exporter/enhanced/enhanced.go:196 +0x139
      github.com/percona/rds_exporter/enhanced.(*Exporter).collectValues.func1(0xc42049b2f0, 0xc42004d200, 0xc42033e7e0, 0xc420017d90, 0xb, 0xc420017d70, 0x9, 0xc420019480, 0x14, 0xc420070540, ...)
      	/builddir/build/BUILD/rds_exporter-a9b0970f99f579ca25b0fa763e413ed8d5ca3b52/src/github.com/percona/rds_exporter/enhanced/enhanced.go:171 +0xdb
      created by github.com/percona/rds_exporter/enhanced.(*Exporter).collectValues
      	/builddir/build/BUILD/rds_exporter-a9b0970f99f579ca25b0fa763e413ed8d5ca3b52/src/github.com/percona/rds_exporter/enhanced/enhanced.go:168 +0x5f9
      

        Smart Checklist

          Attachments

            Issue Links

              Activity

                People

                Assignee:
                Unassigned
                Reporter:
                mykola.marzhan Mykola Marzhan
                Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                  Dates

                  Created:
                  Updated:
                  Resolved:

                    Time Tracking

                    Estimated:
                    Original Estimate - Not Specified
                    Not Specified
                    Remaining:
                    Remaining Estimate - 0 minutes
                    0m
                    Logged:
                    Time Spent - 6 hours
                    6h