Uploaded image for project: 'Percona Monitoring and Management'
  1. Percona Monitoring and Management
  2. PMM-4829

The MongoDB Exporter wasn't able to collect metrics from hidden nodes without either the latest driver or using the 'connect-direct' parameter.

    XMLWordPrintable

    Details

    • Story Points:
      1
    • Sprint:
      Platform Sprint 14
    • Needs Review:
      Yes
    • Needs QA:
      Yes
    • Needs Packaging:
      No
    • Needs Doc:
      Yes

      Description

      Issue:  Mongodb Exporter can't gather metrics from hidden nodes and failing with the following error.

      DEBU[0000] Collecting Replset Status source="mongodb_collector.go:325"ERRO[0000] Failed to get replSet status: (Unauthorized) not authorized on admin to execute command { replSetGetStatus: 1, $db: "admin", $readPreference:
      { mode: "primaryPreferred" }
      } source="replset_status.go:307"DEBU[0000] Collecting Replset Oplog Status source="mongodb_collector.go:331"ERRO[0000] Failed to get oplog collection status: (Unauthorized) not authorized on local to execute command { collStats: "oplog.rs", $db: "local", $readPreference:
      { mode: "primaryPreferred" }
      }
      

       

      Steps to Reproduce:

      Issue reproducible with PMM2 Mongodb_exporter (mongodb exporter v0.10.0) same used in PMM1as well and MongoDB 3.6.16/MongoDB 4.0.14
      pmm2/exporters/mongodb_exporter  

      1. setup mongo replica set with the following settings.

      Mongodb config:

      storage:
       dbPath: /var/lib/mongodb3
      # where to write logging data.
      systemLog:
       destination: file
       logAppend: true
       path: /var/log/mongodb/mongod3.log
      processManagement:
       fork: true
       pidFilePath: /var/run/mongod3.pid
      # network interfaces
      net:
       port: 26003
       bindIp: 127.0.0.1
      replication:
       replSetName: rs0
      security:
       authorization: enabled
       keyFile: /etc/mongo-keyfile

       

      mongodb user privileges/role:

      { "_id" : "admin.mongodb_exporter", "userId" : UUID("634d7642-2cd1-4311-8c79-41cf7f056c92"), "user" : "mongodb_exporter", "db" : "admin", "roles" : [ { "role" : "clusterMonitor", "db" : "admin" }, { "role" : "readAnyDatabase", "db" : "admin" }, { "role" : "read", "db" : "local" }, { "role" : "userAdminAnyDatabase", "db" : "admin" }, { "role" : "root", "db" : "admin" } ] }

      Configure the replicaSet: [ 1 node hidden in replica set ]

      rs0:PRIMARY> cfg = rs.conf()
      {
              "_id" : "rs0",
              "version" : 12,
              "protocolVersion" : NumberLong(1),
              "members" : [
                      {
                              "_id" : 0,
                              "host" : "osboxes:26001",
                              "arbiterOnly" : false,
                              "buildIndexes" : true,
                              "hidden" : false,
                              "priority" : 1,
                              "tags" : {                        },
                              "slaveDelay" : NumberLong(0),
                              "votes" : 1
                      },
                      {
                              "_id" : 1,
                              "host" : "osboxes:26002",
                              "arbiterOnly" : false,
                              "buildIndexes" : true,
                              "hidden" : false,
                              "priority" : 1,
                              "tags" : {                        },
                              "slaveDelay" : NumberLong(0),
                              "votes" : 1
                      },
                      {
                              "_id" : 2,
                              "host" : "osboxes:26003",
                              "arbiterOnly" : false,
                              "buildIndexes" : true,
                              "hidden" : true,
                              "priority" : 0,
                              "tags" : {                        },
                              "slaveDelay" : NumberLong(0),
                              "votes" : 1
                      }
              ],
              "settings" : {
                      "chainingAllowed" : true,
                      "heartbeatIntervalMillis" : 2000,
                      "heartbeatTimeoutSecs" : 10,
                      "electionTimeoutMillis" : 10000,
                      "catchUpTimeoutMillis" : 60000,
                      "catchUpTakeoverDelayMillis" : 30000,
                      "getLastErrorModes" : {                },
                      "getLastErrorDefaults" : {
                              "w" : 1,
                              "wtimeout" : 0
                      },
                      "replicaSetId" : ObjectId("5e2edcb21cf792d4fea4a23d")
              }
      }

       

      Starting the exporter with a hidden node:

      /usr/local/percona/pmm2/exporters/mongodb_exporter --web.listen-address=:42042 --mongodb.uri=mongodb://mongodb_exporter:foobar@osboxes:26003/ --log.level="debug"INFO[0000] Starting mongodb_exporter (version=, branch=, revision=) source="mongodb_exporter.go:94"INFO[0000] Build context (go=go1.12.12, user=, date=19691231-19:00:00) source="mongodb_exporter.go:95"DEBU[0000] Connected to: mongodb://***:***@osboxes:26003/ (node type: replset, server version: 3.6.16) source="mongodb_collector.go:216"DEBU[0000] Collecting Server Status source="mongodb_collector.go:269"ERRO[0000] Failed to get server status: (Unauthorized) not authorized on admin to execute command { serverStatus: 1, recordStats: 0, opLatencies:
      { histograms: true }
      , $db: "admin", $readPreference:
      { mode: "primaryPreferred" }
      } source="server_status.go:151"DEBU[0000] Collecting ReplSetConf Metrics source="mongodb_collector.go:319"ERRO[0000] Failed to get replSetGetConfig: (Unauthorized) not authorized on admin to execute command { replSetGetConfig: 1, $db: "admin", $readPreference:
      { mode: "primaryPreferred" }
      }. source="replset_conf.go:115"DEBU[0000] Collecting Replset Status source="mongodb_collector.go:325"ERRO[0000] Failed to get replSet status: (Unauthorized) not authorized on admin to execute command { replSetGetStatus: 1, $db: "admin", $readPreference:
      { mode: "primaryPreferred" }
      } source="replset_status.go:307"DEBU[0000] Collecting Replset Oplog Status source="mongodb_collector.go:331"ERRO[0000] Failed to get oplog collection status: (Unauthorized) not authorized on local to execute command { collStats: "oplog.rs", $db: "local", $readPreference:
      { mode: "primaryPreferred" }
      } source="oplog_status.go:144"ERRO[0000] Failed to get oplog timestamps status: (Unauthorized) not authorized on local to execute command { find: "oplog.rs", comment: "/tmp/go/src/github.com/percona/mongodb_exporter/collector/mongod/oplog_status.go:85", filter: {}, limit: 1, singleBatch: true, sort: { $natural: -1 }, $db: "local", $readPreference:
      { mode: "nearest" }
      } source="oplog_status.go:149"INFO[0000] Starting HTTP server for http://:42042/metrics ... source="server.go:140"
      

       

      If we remove the hidden flag for the same replica set, mongodb_exporter works fine.
      rs0:PRIMARY> cfg.members[2].hidden = false;falsers0:PRIMARY> rs.reconfig(cfg);

      { "ok" : 1 }

      rs0:PRIMARY>
       

      And start the exporter again: 

      /usr/local/percona/pmm2/exporters/mongodb_exporter --web.listen-address=:42042 --mongodb.uri=mongodb://mongodb_exporter:foobar@osboxes:26003/ --log.level="debug"
      INFO[0000] Starting mongodb_exporter (version=, branch=, revision=) source="mongodb_exporter.go:94"
      INFO[0000] Build context (go=go1.12.12, user=, date=19691231-19:00:00) source="mongodb_exporter.go:95"
      DEBU[0000] Connected to: mongodb://****:****@osboxes:26003/ (node type: replset, server version: 3.6.16) source="mongodb_collector.go:216"
      DEBU[0000] Collecting Server Status source="mongodb_collector.go:269"
      DEBU[0000] Collecting ReplSetConf Metrics source="mongodb_collector.go:319"
      DEBU[0000] Collecting Replset Status source="mongodb_collector.go:325"
      DEBU[0000] Collecting Replset Oplog Status source="mongodb_collector.go:331"
      INFO[0000] Starting HTTP server for http://:42042/metrics ... source="server.go:140"
        

      Expected Result: mongodb_exporter should not fail if the MongoDB replica set has a hidden node. 

       

      --------------------------

       

      Original report:

       

      Hi, 

      mongodb exporter v0.10.0 can't gather metrics from hidden nodes, and logs the following messages:

      mongodb-exporter[6]: time="2019-10-15T15:53:00Z" level=error msg="Failed to get replSetGetConfig: not authorized on admin to execute command { replSetGetConfig: 1 }." source="replset_conf.go:115"
      mongodb-exporter[6]: time="2019-10-15T15:53:00Z" level=error msg="Failed to get replSet status: not authorized on admin to execute command { replSetGetStatus: 1 }" source="replset_status.go:307"
      mongodb-exporter[6]: time="2019-10-15T15:53:00Z" level=error msg="Failed to get oplog collection status: not authorized on local to execute command { collStats: \"oplog.rs\" }" source="oplog_status.go:144"
      mongodb-exporter[6]: time="2019-10-15T15:53:00Z" level=error msg="Failed to get oplog timestamps status: not authorized on local to execute command { find: \"oplog.rs\", comment: \"/home/travis/gopath/src/github.com/percona/mongodb_exporter/collector/mongod/oplog_status.go:85\", filter: {}, limit: 1, singleBatch: true, sort: { $natural: -1 } }" source="oplog_status.go:149"

      MongoDB log sthe following message:

      { find: "oplog.rs", comment: "/home/travis/gopath/src/github.com/percona/mongodb_exporter/collector/mongod/oplog_status.go:85", filter: {}, limit: 1, singleBatch: true, sort: { $natural: -1 } }

       

      Executing queries on local.oplog.rs manually works, plus this only happens on hidden nodes. Exporter on all other nodes works correctly

       

       

       

      -----------------

      The Result

      During the investigation we found out than  to monitor hidden node   user need to pass ?connect=direct  parameter 

      Actions

       

       

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              Unassigned
              Reporter:
              Costa Filipe
              Votes:
              2 Vote for this issue
              Watchers:
              11 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - Not Specified
                  Not Specified
                  Logged:
                  Time Spent - 3 hours, 40 minutes
                  3h 40m

                    Smart Checklist