Uploaded image for project: 'Percona Operator for MongoDB'
  1. Percona Operator for MongoDB
  2. K8SPSMDB-765

pmm client cannot connect to mongodb when requireTLS mode activated

Details

    • Bug
    • Status: Done
    • Medium
    • Resolution: Fixed
    • 1.13.0
    • 1.13.0
    • None
    • Yes
    • Yes

    Description

      If we set net.tls.mode=requireTLS for the cluster pmm client will not be able to connect to mongodb and instances will not be monitored.

      One strange thing is that even though pmm agent is failing the cluster status seems to be ready.

      pmm-client log:

      INFO[2022-09-03T11:31:32.182+00:00]     /usr/local/go/src/net/http/server.go:2916 +0x43b  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-process type=mongodb_exporter
      INFO[2022-09-03T11:31:32.182+00:00] net/http.(*conn).serve(0xc0000315e0, {0xddeba0, 0xc00009ffb0})  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-process type=mongodb_exporter
      INFO[2022-09-03T11:31:32.182+00:00]     /usr/local/go/src/net/http/server.go:1966 +0x5d7  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-process type=mongodb_exporter
      INFO[2022-09-03T11:31:32.182+00:00] created by net/http.(*Server).Serve           agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-process type=mongodb_exporter
      INFO[2022-09-03T11:31:32.182+00:00]     /usr/local/go/src/net/http/server.go:3071 +0x4db  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-process type=mongodb_exporter
      ERRO[2022-09-03T11:31:35.184+00:00] time="2022-09-03T11:31:35Z" level=error msg="Cannot connect to MongoDB: cannot connect to MongoDB: server selection error: server selection timeout, current topolog
      y: { Type: Single, Servers: [{ Addr: localhost:27017, Type: Unknown, Last error: connection() error occurred during connection handshake: connection(localhost:27017[-734]) socket was unexpectedly clos
      ed: EOF }, ] }"  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-process type=mongodb_exporter
      INFO[2022-09-03T11:31:35.185+00:00] 2022/09/03 11:31:35 http: panic serving 127.0.0.1:51994: runtime error: invalid memory address or nil pointer dereference  agentID=/agent_id/89de40b0-d5ad-4df3-ba33
      -e172398a6154 component=agent-process type=mongodb_exporter
      INFO[2022-09-03T11:31:35.185+00:00] goroutine 4199 [running]:                     agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-process type=mongodb_exporter
      INFO[2022-09-03T11:31:35.185+00:00] net/http.(*conn).serve.func1()                agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-process type=mongodb_exporter
      INFO[2022-09-03T11:31:35.185+00:00]     /usr/local/go/src/net/http/server.go:1825 +0xbf  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-process type=mongodb_exporter
      INFO[2022-09-03T11:31:35.185+00:00] panic({0xb5e840, 0x1247910})                  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-process type=mongodb_exporter
      INFO[2022-09-03T11:31:35.185+00:00]     /usr/local/go/src/runtime/panic.go:844 +0x258  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-process type=mongodb_exporter
      INFO[2022-09-03T11:31:35.185+00:00] go.mongodb.org/mongo-driver/mongo.newDatabase(0x0, {0xc246e2, 0x5}, {0x0?, 0x30?, 0x1261880?})  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=age
      nt-process type=mongodb_exporter
      INFO[2022-09-03T11:31:35.185+00:00]     /tmp/go/pkg/mod/go.mongodb.org/[email protected]/mongo/database.go:47 +0xcd  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-process ty
      pe=mongodb_exporter
      INFO[2022-09-03T11:31:35.185+00:00] go.mongodb.org/mongo-driver/mongo.(*Client).Database(...)  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-process type=mongodb_exporter
      INFO[2022-09-03T11:31:35.185+00:00]     /tmp/go/pkg/mod/go.mongodb.org/[email protected]/mongo/client.go:837  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-process type=mong
      odb_exporter
      INFO[2022-09-03T11:31:35.185+00:00] github.com/percona/mongodb_exporter/exporter.getClusterRole({0xddeb68, 0xc0003c9380}, 0xc000433508?)  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 compone
      nt=agent-process type=mongodb_exporter
      INFO[2022-09-03T11:31:35.185+00:00]     /tmp/go/src/github.com/percona/mongodb_exporter/exporter/topology_info.go:172 +0x8b  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-proc
      ess type=mongodb_exporter
      INFO[2022-09-03T11:31:35.185+00:00] github.com/percona/mongodb_exporter/exporter.(*topologyInfo).loadLabels(0xc000459380, {0xddeb68, 0xc0003c9380})  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6
      154 component=agent-process type=mongodb_exporter
      INFO[2022-09-03T11:31:35.185+00:00]     /tmp/go/src/github.com/percona/mongodb_exporter/exporter/topology_info.go:105 +0xeb  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-proc
      ess type=mongodb_exporter
      INFO[2022-09-03T11:31:35.185+00:00] github.com/percona/mongodb_exporter/exporter.newTopologyInfo({0xddeb68, 0xc0003c9380}, 0x0, 0xc00032cd90)  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 co
      mponent=agent-process type=mongodb_exporter
      INFO[2022-09-03T11:31:35.185+00:00]     /tmp/go/src/github.com/percona/mongodb_exporter/exporter/topology_info.go:75 +0xb0  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-proce
      ss type=mongodb_exporter
      INFO[2022-09-03T11:31:35.185+00:00] github.com/percona/mongodb_exporter/exporter.(*Exporter).Handler.func1({0xdde4c0, 0xc0003302a0}, 0xc0000f8400)  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a61
      54 component=agent-process type=mongodb_exporter
      INFO[2022-09-03T11:31:35.185+00:00]     /tmp/go/src/github.com/percona/mongodb_exporter/exporter/exporter.go:304 +0x39f  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-process
      type=mongodb_exporter
      INFO[2022-09-03T11:31:35.185+00:00] net/http.HandlerFunc.ServeHTTP(0xc0001a0540?, {0xdde4c0?, 0xc0003302a0?}, 0xda?)  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-process typ
      e=mongodb_exporter
      INFO[2022-09-03T11:31:35.185+00:00]     /usr/local/go/src/net/http/server.go:2084 +0x2f  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-process type=mongodb_exporter
      INFO[2022-09-03T11:31:35.185+00:00] net/http.(*ServeMux).ServeHTTP(0xb58d40?, {0xdde4c0, 0xc0003302a0}, 0xc0000f8400)  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-process ty
      pe=mongodb_exporter
      INFO[2022-09-03T11:31:35.185+00:00]     /usr/local/go/src/net/http/server.go:2462 +0x149  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-process type=mongodb_exporter
      INFO[2022-09-03T11:31:35.185+00:00] github.com/prometheus/exporter-toolkit/web.(*webHandler).ServeHTTP(0xc0000cc5c0, {0xdde4c0?, 0xc0003302a0}, 0xc000200000?)  agentID=/agent_id/89de40b0-d5ad-4df3-ba3
      3-e172398a6154 component=agent-process type=mongodb_exporter
      INFO[2022-09-03T11:31:35.185+00:00]     /tmp/go/pkg/mod/github.com/prometheus/[email protected]/web/handler.go:130 +0x896  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-
      process type=mongodb_exporter
      INFO[2022-09-03T11:31:35.185+00:00] net/http.serverHandler.ServeHTTP({0xc00034bc20?}, {0xdde4c0, 0xc0003302a0}, 0xc0000f8400)  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-pr
      ocess type=mongodb_exporter
      INFO[2022-09-03T11:31:35.185+00:00]     /usr/local/go/src/net/http/server.go:2916 +0x43b  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-process type=mongodb_exporter
      INFO[2022-09-03T11:31:35.185+00:00] net/http.(*conn).serve(0xc0001b0640, {0xddeba0, 0xc00009ffb0})  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-process type=mongodb_exporter
      INFO[2022-09-03T11:31:35.185+00:00]     /usr/local/go/src/net/http/server.go:1966 +0x5d7  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-process type=mongodb_exporter
      INFO[2022-09-03T11:31:35.185+00:00] created by net/http.(*Server).Serve           agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-process type=mongodb_exporter
      INFO[2022-09-03T11:31:35.185+00:00]     /usr/local/go/src/net/http/server.go:3071 +0x4db  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-process type=mongodb_exporter
      INFO[2022-09-03T11:31:35.185+00:00] 2022-09-03T11:31:35.185Z    error   VictoriaMetrics/lib/promscrape/scrapework.go:355        error when scraping "http://127.0.0.1:30102/metrics" from job "mongodb_e
      xporter_agent_id_89de40b0-d5ad-4df3-ba33-e172398a6154_hr" with labels {agent_id="/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154",agent_type="mongodb_exporter",cluster="my-cluster-name",instance="/agen
      t_id/89de40b0-d5ad-4df3-ba33-e172398a6154",job="mongodb_exporter_agent_id_89de40b0-d5ad-4df3-ba33-e172398a6154_hr",node_id="/node_id/e35fd660-af44-4aee-8d42-6aa424acb14d",node_name="test-my-cluster-na
      me-rs0-0",node_type="container",service_id="/service_id/101def6a-9e8a-49f6-a3c7-cc4739e4699e",service_name="test-my-cluster-name-rs0-0",service_type="mongodb"}: error when scraping "http://127.0.0.1:3
      0102/metrics": the server closes all the connection attempts: the server closed connection before returning the first response byte. Make sure the server returns 'Connection: close' response header be
      fore closing the connection  agentID=/agent_id/1f15ab06-5e41-4a2e-81b6-a7bab5817156 component=agent-process type=vm_agent
      

      Pods:

      NAME              ENDPOINT                                        STATUS   AGE
      my-cluster-name   my-cluster-name-mongos.test.svc.cluster.local   ready    9m41s
      
      NAME                                               READY   STATUS    RESTARTS        AGE
      my-cluster-name-cfg-0                              3/3     Running   0               9m41s
      my-cluster-name-cfg-1                              3/3     Running   1 (7m51s ago)   9m5s
      my-cluster-name-cfg-2                              3/3     Running   1 (7m51s ago)   8m31s
      my-cluster-name-mongos-0                           2/2     Running   0               8m17s
      my-cluster-name-mongos-1                           2/2     Running   0               8m1s
      my-cluster-name-mongos-2                           2/2     Running   0               7m47s
      my-cluster-name-rs0-0                              3/3     Running   0               9m41s
      my-cluster-name-rs0-1                              3/3     Running   0               9m8s
      my-cluster-name-rs0-2                              3/3     Running   0               8m37s
      percona-server-mongodb-operator-6677c8cbf7-r4wmq   1/1     Running   0               10m
      pmm-0                                              1/1     Running   0               19m
      

      Attachments

        Issue Links

          Activity

            People

              ege.gunes Ege Gunes
              tomislav.plavcic@percona.com Tomislav Plavcic
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Smart Checklist