Uploaded image for project: 'Percona Monitoring and Management'
  1. Percona Monitoring and Management
  2. PMM-642

pmm-client 1.1.1 mongodb_exporter dies after mongod restart, username+password logged in errors

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Priority: High
    • Resolution: Done
    • Affects Version/s: 1.1.1
    • Fix Version/s: 1.1.2
    • Component/s: PMM Client
    • Labels:
      None
    • Environment:

      Any pmm-client version + mongodb:metrics

      Description

      If mongodb is restarted after the mongodb:metrics service is started, the mongodb_exporter process crashes and does not recover. The mongodb:metrics service must be restarted to fix this.

      This is due to a function call on a null variable when no connection is possible in percona/mongodb_exporter.

      This is resolved by this GitHub pull request: https://github.com/percona/mongodb_exporter/pull/55.

      This also resolves this GitHub issue: https://github.com/percona/mongodb_exporter/issues/50.

      Lastly, while investigating a report of this error from a user I noticed we print usernames/passwords for the database user when auth is used. I fixed this in the same pull-request as well as a connection timeout that is too high, reduced from 10 sec -> 5 sec to stop piling-up Prometheus polls. The exporter should be contacting localhost so 5 seconds is very long.

      Error after mongod restart/stop, pmm-client 1.0.7 (same thing on 1.1.1):

      ### Warning: the exporter is in beta/experimental state and field names are very 
      ### likely to change in the future and features may change or get removed! 
      ### See: https://github.com/Percona-Lab/prometheus_mongodb_exporter for updates 
      mongodb_exporter version: 1.0.7, git commit hash: 2177e1a7b7dd2335533322b547d7041e6d25c1c7 
      E0109 18:46:13.518864 3588 connection.go:27] Cannot connect to server using url mongodb://REDACTED@localhost:27017/admin: no reachable servers 
      panic: runtime error: invalid memory address or nil pointer dereference 
      [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x67991e]
      
      goroutine 12 [running]: 
      panic(0x7592e0, 0xc42000e0e0) 
      /usr/local/go1.7.4/src/runtime/panic.go:500 +0x1a1 
      github.com/Percona-Lab/prometheus_mongodb_exporter/vendor/gopkg.in/mgo%2ev2.(*Session).Close(0x0) 
      /mnt/workspace/pmm-client-tarball/pmm-client-1.0.7/src/github.com/Percona-Lab/prometheus_mongodb_exporter/vendor/gopkg.in/mgo.v2/session.go:1557 +0x2e 
      github.com/Percona-Lab/prometheus_mongodb_exporter/collector.(*MongodbCollector).Describe(0xc42016e090, 0xc420082d80) 
      /mnt/workspace/pmm-client-tarball/pmm-client-1.0.7/src/github.com/Percona-Lab/prometheus_mongodb_exporter/collector/mongodb_collector.go:47 +0xef 
      github.com/Percona-Lab/prometheus_mongodb_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*Registry).Register.func1(0x9589c0, 0xc42016e090, 0xc420082d80) 
      /mnt/workspace/pmm-client-tarball/pmm-client-1.0.7/src/github.com/Percona-Lab/prometheus_mongodb_exporter/vendor/github.com/prometheus/client_golang/prometheus/registry.go:292 +0x3b 
      created by github.com/Percona-Lab/prometheus_mongodb_exporter/vendor/github.com/prometheus/client_golang/prometheus.(*Registry).Register 
      /mnt/workspace/pmm-client-tarball/pmm-client-1.0.7/src/github.com/Percona-Lab/prometheus_mongodb_exporter/vendor/github.com/prometheus/client_golang/prometheus/registry.go:294 +0x11e
      

      Test with pull-request changes (mongodb restarted in the middle of output below - no crash!):

      I0309 00:54:20.719320   21709 mongodb_collector.go:64] Connected to: mongodb://****:****@localhost:27017 (node type: replset, server version: 3.2.11-3.1)
      I0309 00:54:20.719336   21709 mongodb_collector.go:93] Collecting Server Status
      I0309 00:54:20.721113   21709 mongodb_collector.go:103] Collecting Replset Status
      I0309 00:54:20.721910   21709 mongodb_collector.go:109] Collecting Replset Oplog Status
      
      I0309 00:54:21.771215   21709 mongodb_collector.go:64] Connected to: mongodb://****:****@localhost:27017 (node type: replset, server version: 3.2.11-3.1)
      I0309 00:54:21.771233   21709 mongodb_collector.go:93] Collecting Server Status
      I0309 00:54:21.773353   21709 mongodb_collector.go:103] Collecting Replset Status
      I0309 00:54:21.774629   21709 mongodb_collector.go:109] Collecting Replset Oplog Status
      
      E0309 00:54:28.003026   21709 connection.go:42] Cannot connect to server using url mongodb://****:****@localhost:27017: no reachable servers
      
      E0309 00:54:34.569117   21709 connection.go:42] Cannot connect to server using url mongodb://****:****@localhost:27017: no reachable servers
      
      I0309 00:54:40.648456   21709 mongodb_collector.go:64] Connected to: mongodb://****:****@localhost:27017 (node type: replset, server version: 3.2.11-3.1)
      I0309 00:54:40.648482   21709 mongodb_collector.go:93] Collecting Server Status
      I0309 00:54:40.651508   21709 mongodb_collector.go:103] Collecting Replset Status
      I0309 00:54:40.652532   21709 mongodb_collector.go:109] Collecting Replset Oplog Status
      
      I0309 00:54:41.710551   21709 mongodb_collector.go:64] Connected to: mongodb://****:****@localhost:27017 (node type: replset, server version: 3.2.11-3.1)
      I0309 00:54:41.710571   21709 mongodb_collector.go:93] Collecting Server Status
      I0309 00:54:41.712038   21709 mongodb_collector.go:103] Collecting Replset Status
      I0309 00:54:41.713355   21709 mongodb_collector.go:109] Collecting Replset Oplog Status
      

      Notice the exporter recovers when the mongodb instance comes back up. Cannot connect errors now show in 5 seconds, not 10. Also username/passwords are now obfuscated.

        Smart Checklist

          Attachments

            Activity

              People

              • Assignee:
                roma.novikov Roma Novikov
                Reporter:
                tim.vaillancourt Tim Vaillancourt (Inactive)
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: