Replica sets can't connect to config server when TLS is used

I've got a MongoDB 4.0.11 cluster set up with routers, a config server, and two replicasets. Strict SSL checking is used, but I've got the same set of certs on each node. pbm-agent is configured using the URI

PBM_MONGODB_URI=mongodb://pbmAgent:[email protected]:27017/?authSource=admin&replicaSet=mongo1&tls=true&tlsCertificateKeyFile=/data/db/private/client.pem&tlsCAFile=/data/db/private/cacert.pem

and can talk to the local mongod process. pbm is also talking to the config servers. When I execute the pbm backup command, the backup hangs. Looking at the logs, pdb-agent is failing to connect to the config server:

2019/11/22 23:37:20 connect to mongodb: create mongo connection to configsvr: mongo ping: server selection error: server selection timeout

info current topology: Type: Unknown
info Servers:
Addr: mongocnf-cmgo-replica-0.mongocnf-cmgo.default.svc.cluster.local:27017, Type: Unknown, State: Connected, Average RTT: 0, Last error: connection() : connection(mongocnf-cmgo-replica-0.mongocnf-cmgo.default.svc.cluster.local:27017[-187]) unable to decode message length: EOF
23:37:20 docker-entrypoint.sh: LOG:info Addr: mongocnf-cmgo-replica-1.mongocnf-cmgo.default.svc.cluster.local:27017, Type: Unknown, State: Connected, Average RTT: 0, Last error: connection() : connection(mongocnf-cmgo-replica-1.mongocnf-cmgo.default.svc.cluster.local:27017[-186]) unable to decode message length: EOF
23:37:20 docker-entrypoint.sh: LOG:info Addr: mongocnf-cmgo-replica-2.mongocnf-cmgo.default.svc.cluster.local:27017, Type: Unknown, State: Connected, Average RTT: 0, Last error: connection() : connection(mongocnf-cmgo-replica-2.mongocnf-cmgo.default.svc.cluster.local:27017[-185]) unable to decode message length: EOF

On the Mongod process on the client is complaining that the connections are not SSL:

I NETWORK [conn9279] Error receiving request from client: SSLHandshakeFailed: The server is configured to only allow SSL connections. Ending connection from 192.168.1.198:60382 (connection id: 9279)

192.168.1.198 is the address of the mongo replica that is doing part of the backup. The other replica set has a node going crazy this way as well.

I don't see anything in the documentation about how to configure the SSL from the replicas to the config server. If they just used the URI that the PBM backup command was given, it would work.