Uploaded image for project: 'Percona Operator for MongoDB'
  1. Percona Operator for MongoDB
  2. K8SPSMDB-765

pmm client cannot connect to mongodb when requireTLS mode activated

    XMLWordPrintable

Details

    • Bug
    • Status: Done
    • Medium
    • Resolution: Fixed
    • 1.13.0
    • 1.13.0
    • None
    • Yes
    • Yes

    Description

      If we set net.tls.mode=requireTLS for the cluster pmm client will not be able to connect to mongodb and instances will not be monitored.

      One strange thing is that even though pmm agent is failing the cluster status seems to be ready.

      pmm-client log:

      INFO[2022-09-03T11:31:32.182+00:00]     /usr/local/go/src/net/http/server.go:2916 +0x43b  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-process type=mongodb_exporter
INFO[2022-09-03T11:31:32.182+00:00] net/http.(*conn).serve(0xc0000315e0, {0xddeba0, 0xc00009ffb0})  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-process type=mongodb_exporter
INFO[2022-09-03T11:31:32.182+00:00]     /usr/local/go/src/net/http/server.go:1966 +0x5d7  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-process type=mongodb_exporter
INFO[2022-09-03T11:31:32.182+00:00] created by net/http.(*Server).Serve           agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-process type=mongodb_exporter
INFO[2022-09-03T11:31:32.182+00:00]     /usr/local/go/src/net/http/server.go:3071 +0x4db  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-process type=mongodb_exporter
ERRO[2022-09-03T11:31:35.184+00:00] time="2022-09-03T11:31:35Z" level=error msg="Cannot connect to MongoDB: cannot connect to MongoDB: server selection error: server selection timeout, current topolog
y: { Type: Single, Servers: [{ Addr: localhost:27017, Type: Unknown, Last error: connection() error occurred during connection handshake: connection(localhost:27017[-734]) socket was unexpectedly clos
ed: EOF }, ] }"  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-process type=mongodb_exporter
INFO[2022-09-03T11:31:35.185+00:00] 2022/09/03 11:31:35 http: panic serving 127.0.0.1:51994: runtime error: invalid memory address or nil pointer dereference  agentID=/agent_id/89de40b0-d5ad-4df3-ba33
-e172398a6154 component=agent-process type=mongodb_exporter
INFO[2022-09-03T11:31:35.185+00:00] goroutine 4199 [running]:                     agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-process type=mongodb_exporter
INFO[2022-09-03T11:31:35.185+00:00] net/http.(*conn).serve.func1()                agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-process type=mongodb_exporter
INFO[2022-09-03T11:31:35.185+00:00]     /usr/local/go/src/net/http/server.go:1825 +0xbf  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-process type=mongodb_exporter
INFO[2022-09-03T11:31:35.185+00:00] panic({0xb5e840, 0x1247910})                  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-process type=mongodb_exporter
INFO[2022-09-03T11:31:35.185+00:00]     /usr/local/go/src/runtime/panic.go:844 +0x258  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-process type=mongodb_exporter
INFO[2022-09-03T11:31:35.185+00:00] go.mongodb.org/mongo-driver/mongo.newDatabase(0x0, {0xc246e2, 0x5}, {0x0?, 0x30?, 0x1261880?})  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=age
nt-process type=mongodb_exporter
INFO[2022-09-03T11:31:35.185+00:00]     /tmp/go/pkg/mod/go.mongodb.org/[email protected]/mongo/database.go:47 +0xcd  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-process ty
pe=mongodb_exporter
INFO[2022-09-03T11:31:35.185+00:00] go.mongodb.org/mongo-driver/mongo.(*Client).Database(...)  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-process type=mongodb_exporter
INFO[2022-09-03T11:31:35.185+00:00]     /tmp/go/pkg/mod/go.mongodb.org/[email protected]/mongo/client.go:837  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-process type=mong
odb_exporter
INFO[2022-09-03T11:31:35.185+00:00] github.com/percona/mongodb_exporter/exporter.getClusterRole({0xddeb68, 0xc0003c9380}, 0xc000433508?)  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 compone
nt=agent-process type=mongodb_exporter
INFO[2022-09-03T11:31:35.185+00:00]     /tmp/go/src/github.com/percona/mongodb_exporter/exporter/topology_info.go:172 +0x8b  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-proc
ess type=mongodb_exporter
INFO[2022-09-03T11:31:35.185+00:00] github.com/percona/mongodb_exporter/exporter.(*topologyInfo).loadLabels(0xc000459380, {0xddeb68, 0xc0003c9380})  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6
154 component=agent-process type=mongodb_exporter
INFO[2022-09-03T11:31:35.185+00:00]     /tmp/go/src/github.com/percona/mongodb_exporter/exporter/topology_info.go:105 +0xeb  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-proc
ess type=mongodb_exporter
INFO[2022-09-03T11:31:35.185+00:00] github.com/percona/mongodb_exporter/exporter.newTopologyInfo({0xddeb68, 0xc0003c9380}, 0x0, 0xc00032cd90)  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 co
mponent=agent-process type=mongodb_exporter
INFO[2022-09-03T11:31:35.185+00:00]     /tmp/go/src/github.com/percona/mongodb_exporter/exporter/topology_info.go:75 +0xb0  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-proce
ss type=mongodb_exporter
INFO[2022-09-03T11:31:35.185+00:00] github.com/percona/mongodb_exporter/exporter.(*Exporter).Handler.func1({0xdde4c0, 0xc0003302a0}, 0xc0000f8400)  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a61
54 component=agent-process type=mongodb_exporter
INFO[2022-09-03T11:31:35.185+00:00]     /tmp/go/src/github.com/percona/mongodb_exporter/exporter/exporter.go:304 +0x39f  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-process
type=mongodb_exporter
INFO[2022-09-03T11:31:35.185+00:00] net/http.HandlerFunc.ServeHTTP(0xc0001a0540?, {0xdde4c0?, 0xc0003302a0?}, 0xda?)  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-process typ
e=mongodb_exporter
INFO[2022-09-03T11:31:35.185+00:00]     /usr/local/go/src/net/http/server.go:2084 +0x2f  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-process type=mongodb_exporter
INFO[2022-09-03T11:31:35.185+00:00] net/http.(*ServeMux).ServeHTTP(0xb58d40?, {0xdde4c0, 0xc0003302a0}, 0xc0000f8400)  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-process ty
pe=mongodb_exporter
INFO[2022-09-03T11:31:35.185+00:00]     /usr/local/go/src/net/http/server.go:2462 +0x149  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-process type=mongodb_exporter
INFO[2022-09-03T11:31:35.185+00:00] github.com/prometheus/exporter-toolkit/web.(*webHandler).ServeHTTP(0xc0000cc5c0, {0xdde4c0?, 0xc0003302a0}, 0xc000200000?)  agentID=/agent_id/89de40b0-d5ad-4df3-ba3
3-e172398a6154 component=agent-process type=mongodb_exporter
INFO[2022-09-03T11:31:35.185+00:00]     /tmp/go/pkg/mod/github.com/prometheus/[email protected]/web/handler.go:130 +0x896  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-
process type=mongodb_exporter
INFO[2022-09-03T11:31:35.185+00:00] net/http.serverHandler.ServeHTTP({0xc00034bc20?}, {0xdde4c0, 0xc0003302a0}, 0xc0000f8400)  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-pr
ocess type=mongodb_exporter
INFO[2022-09-03T11:31:35.185+00:00]     /usr/local/go/src/net/http/server.go:2916 +0x43b  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-process type=mongodb_exporter
INFO[2022-09-03T11:31:35.185+00:00] net/http.(*conn).serve(0xc0001b0640, {0xddeba0, 0xc00009ffb0})  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-process type=mongodb_exporter
INFO[2022-09-03T11:31:35.185+00:00]     /usr/local/go/src/net/http/server.go:1966 +0x5d7  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-process type=mongodb_exporter
INFO[2022-09-03T11:31:35.185+00:00] created by net/http.(*Server).Serve           agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-process type=mongodb_exporter
INFO[2022-09-03T11:31:35.185+00:00]     /usr/local/go/src/net/http/server.go:3071 +0x4db  agentID=/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154 component=agent-process type=mongodb_exporter
INFO[2022-09-03T11:31:35.185+00:00] 2022-09-03T11:31:35.185Z    error   VictoriaMetrics/lib/promscrape/scrapework.go:355        error when scraping "http://127.0.0.1:30102/metrics" from job "mongodb_e
xporter_agent_id_89de40b0-d5ad-4df3-ba33-e172398a6154_hr" with labels {agent_id="/agent_id/89de40b0-d5ad-4df3-ba33-e172398a6154",agent_type="mongodb_exporter",cluster="my-cluster-name",instance="/agen
t_id/89de40b0-d5ad-4df3-ba33-e172398a6154",job="mongodb_exporter_agent_id_89de40b0-d5ad-4df3-ba33-e172398a6154_hr",node_id="/node_id/e35fd660-af44-4aee-8d42-6aa424acb14d",node_name="test-my-cluster-na
me-rs0-0",node_type="container",service_id="/service_id/101def6a-9e8a-49f6-a3c7-cc4739e4699e",service_name="test-my-cluster-name-rs0-0",service_type="mongodb"}: error when scraping "http://127.0.0.1:3
0102/metrics": the server closes all the connection attempts: the server closed connection before returning the first response byte. Make sure the server returns 'Connection: close' response header be
fore closing the connection  agentID=/agent_id/1f15ab06-5e41-4a2e-81b6-a7bab5817156 component=agent-process type=vm_agent

      Pods:

      NAME              ENDPOINT                                        STATUS   AGE
my-cluster-name   my-cluster-name-mongos.test.svc.cluster.local   ready    9m41s

NAME                                               READY   STATUS    RESTARTS        AGE
my-cluster-name-cfg-0                              3/3     Running   0               9m41s
my-cluster-name-cfg-1                              3/3     Running   1 (7m51s ago)   9m5s
my-cluster-name-cfg-2                              3/3     Running   1 (7m51s ago)   8m31s
my-cluster-name-mongos-0                           2/2     Running   0               8m17s
my-cluster-name-mongos-1                           2/2     Running   0               8m1s
my-cluster-name-mongos-2                           2/2     Running   0               7m47s
my-cluster-name-rs0-0                              3/3     Running   0               9m41s
my-cluster-name-rs0-1                              3/3     Running   0               9m8s
my-cluster-name-rs0-2                              3/3     Running   0               8m37s
percona-server-mongodb-operator-6677c8cbf7-r4wmq   1/1     Running   0               10m
pmm-0                                              1/1     Running   0               19m

      Attachments

        Issue Links

          relates to

          Improvement - An improvement or enhancement to an existing feature or task. K8SPSMDB-515 Allow setting requireTLS mode for MongoDB through the Operator

          • Medium - Minor loss of function, or other problem where easy workaround is present.
          • Done

          Activity

            People

              ege.gunes Ege Gunes
              tomislav.plavcic@percona.com Tomislav Plavcic
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Smart Checklist