Details

    • Bug
    • Status: Done
    • Medium
    • Resolution: Fixed
    • 1.13.0
    • 1.14.0
    • None
    • Yes
    • Yes

    Description

      Not all messages in the logs contain namespace so sometimes it's unclear which cluster was in question.

      I had 3 clusters:

      • test/my-cluster-name
      • test/my-cluster-name2
      • test2/my-cluster-name
        and it's hard to know which cluster is in question.

      Here's an example:

      2022-09-05T18:52:00.321Z        INFO    controller_psmdb        Created a new mongo key {"Request.Namespace": "test", "Request.Name": "my-cluster-name", "KeyName": "my-cluster-name-mongodb-keyfile"}
      2022-09-05T18:52:00.328Z        INFO    controller_psmdb        Created a new mongo key {"Request.Namespace": "test", "Request.Name": "my-cluster-name", "KeyName": "my-cluster-name-mongodb-encryption-
      key"}
      2022-09-05T18:52:00.717Z        INFO    controller_psmdb        Waiting for the pods    {"replset": "rs0", "size": 3, "pods": 0}
      2022-09-05T18:52:00.734Z        INFO    controller_psmdb        add new job     {"name": "ensure-version/test/my-cluster-name", "schedule": "0 2 * * *"}
      2022-09-05T18:52:00.734Z        INFO    controller_psmdb        Cluster state changed   {"previous": "", "current": "initializing"}
      2022-09-05T18:52:00.867Z        INFO    controller_psmdb        Waiting for the pods    {"replset": "rs0", "size": 3, "pods": 1}
      2022-09-05T18:52:05.871Z        INFO    controller_psmdb        Waiting for the pods    {"replset": "rs0", "size": 3, "pods": 1}
      ...
      2022-09-05T18:52:57.151Z        INFO    controller_psmdb        Waiting for the pods    {"replset": "rs0", "size": 3, "pods": 2}
      2022-09-05T18:53:02.273Z        INFO    controller_psmdb        Waiting for the pods    {"replset": "rs0", "size": 3, "pods": 2}
      2022-09-05T18:53:17.397Z        INFO    controller_psmdb        initiating replset      {"replset": "rs0", "pod": "my-cluster-name-rs0-0"}
      2022-09-05T18:53:23.590Z        INFO    controller_psmdb        replset initialized     {"replset": "rs0", "pod": "my-cluster-name-rs0-0"}
      2022-09-05T18:53:24.500Z        INFO    controller_psmdb        Fixing member tags      {"replset": "rs0"}
      2022-09-05T18:53:24.503Z        INFO    controller_psmdb        Adding new nodes        {"replset": "rs0"}
      2022-09-05T18:53:24.549Z        INFO    controller_psmdb        Configuring member votes and priorities {"replset": "rs0"}
      2022-09-05T18:53:29.496Z        INFO    controller_psmdb        Adding new nodes        {"replset": "rs0"}
      2022-09-05T18:53:29.545Z        INFO    controller_psmdb        Configuring member votes and priorities {"replset": "rs0"}
      2022-09-05T18:53:47.228Z        INFO    controller_psmdb        Cluster state changed   {"previous": "initializing", "current": "ready"}
      2022-09-05T18:53:47.482Z        INFO    controller_psmdb        update Mongo version to 5.0.11-10 (fetched from db)
      2022-09-05T19:00:23.132Z        INFO    controller_psmdb        Created a new mongo key {"Request.Namespace": "test", "Request.Name": "my-cluster-name2", "KeyName": "my-cluster-name2-mongodb-keyfile"}
      2022-09-05T19:00:23.136Z        INFO    controller_psmdb        Created a new mongo key {"Request.Namespace": "test", "Request.Name": "my-cluster-name2", "KeyName": "my-cluster-name-mongodb-encryption
      -key2"}
      2022-09-05T19:00:23.234Z        INFO    controller_psmdb        Waiting for the pods    {"replset": "rs0", "size": 3, "pods": 0}
      2022-09-05T19:00:23.298Z        INFO    controller_psmdb        add new job     {"name": "ensure-version/test/my-cluster-name2", "schedule": "0 2 * * *"}
      2022-09-05T19:00:23.298Z        INFO    controller_psmdb        Cluster state changed   {"previous": "", "current": "initializing"}
      2022-09-05T19:00:23.462Z        INFO    controller_psmdb        Waiting for the pods    {"replset": "rs0", "size": 3, "pods": 1}
      2022-09-05T19:00:28.464Z        INFO    controller_psmdb        Waiting for the pods    {"replset": "rs0", "size": 3, "pods": 1}
      ...
      2022-09-05T19:01:24.952Z        INFO    controller_psmdb        Waiting for the pods    {"replset": "rs0", "size": 3, "pods": 2}
      2022-09-05T19:01:30.088Z        INFO    controller_psmdb        Waiting for the pods    {"replset": "rs0", "size": 3, "pods": 2}
      2022-09-05T19:01:45.215Z        INFO    controller_psmdb        initiating replset      {"replset": "rs0", "pod": "my-cluster-name2-rs0-0"}
      2022-09-05T19:01:51.195Z        INFO    controller_psmdb        replset initialized     {"replset": "rs0", "pod": "my-cluster-name2-rs0-0"}
      2022-09-05T19:01:52.325Z        INFO    controller_psmdb        Fixing member tags      {"replset": "rs0"}
      2022-09-05T19:01:52.328Z        INFO    controller_psmdb        Adding new nodes        {"replset": "rs0"}
      2022-09-05T19:01:52.362Z        INFO    controller_psmdb        Configuring member votes and priorities {"replset": "rs0"}
      2022-09-05T19:01:57.057Z        INFO    controller_psmdb        Adding new nodes        {"replset": "rs0"}
      2022-09-05T19:01:57.090Z        INFO    controller_psmdb        Configuring member votes and priorities {"replset": "rs0"}
      2022-09-05T19:02:09.631Z        INFO    controller_psmdb        Cluster state changed   {"previous": "initializing", "current": "ready"}
      2022-09-05T19:02:10.106Z        INFO    controller_psmdb        update Mongo version to 5.0.11-10 (fetched from db)
      2022-09-05T19:03:39.001Z        INFO    controller_psmdb        Created a new mongo key {"Request.Namespace": "test2", "Request.Name": "my-cluster-name", "KeyName": "my-cluster-name-mongodb-keyfile"}
      2022-09-05T19:03:39.039Z        INFO    controller_psmdb        Created a new mongo key {"Request.Namespace": "test2", "Request.Name": "my-cluster-name", "KeyName": "my-cluster-name-mongodb-encryption-key"}
      2022-09-05T19:03:39.343Z        INFO    controller_psmdb        Waiting for the pods    {"replset": "rs0", "size": 3, "pods": 0}
      2022-09-05T19:03:39.482Z        INFO    controller_psmdb        add new job     {"name": "ensure-version/test2/my-cluster-name", "schedule": "0 2 * * *"}
      2022-09-05T19:03:39.483Z        INFO    controller_psmdb        Cluster state changed   {"previous": "", "current": "initializing"}
      2022-09-05T19:03:39.656Z        INFO    controller_psmdb        Waiting for the pods    {"replset": "rs0", "size": 3, "pods": 1}
      2022-09-05T19:03:44.654Z        INFO    controller_psmdb        Waiting for the pods    {"replset": "rs0", "size": 3, "pods": 1}
      ...
      2022-09-05T19:04:41.052Z        INFO    controller_psmdb        Waiting for the pods    {"replset": "rs0", "size": 3, "pods": 2}
      2022-09-05T19:04:46.178Z        INFO    controller_psmdb        Waiting for the pods    {"replset": "rs0", "size": 3, "pods": 2}
      2022-09-05T19:05:01.305Z        INFO    controller_psmdb        initiating replset      {"replset": "rs0", "pod": "my-cluster-name-rs0-0"}
      2022-09-05T19:05:07.351Z        INFO    controller_psmdb        replset initialized     {"replset": "rs0", "pod": "my-cluster-name-rs0-0"}
      2022-09-05T19:05:08.484Z        INFO    controller_psmdb        Fixing member tags      {"replset": "rs0"}
      2022-09-05T19:05:08.487Z        INFO    controller_psmdb        Adding new nodes        {"replset": "rs0"}
      2022-09-05T19:05:08.505Z        INFO    controller_psmdb        Configuring member votes and priorities {"replset": "rs0"}
      2022-09-05T19:05:13.170Z        INFO    controller_psmdb        Adding new nodes        {"replset": "rs0"}
      2022-09-05T19:05:13.199Z        INFO    controller_psmdb        Configuring member votes and priorities {"replset": "rs0"}
      2022-09-05T19:05:30.886Z        INFO    controller_psmdb        Cluster state changed   {"previous": "initializing", "current": "ready"}
      2022-09-05T19:05:31.296Z        INFO    controller_psmdb        update Mongo version to 5.0.11-10 (fetched from db)
      2022-09-05T19:08:04.262Z        INFO    controller_psmdb        StatefulSet is not up to date   {"sts": "my-cluster-name-rs0"}
      2022-09-05T19:08:04.262Z        INFO    controller_psmdb        StatefulSet is not up to date   {"sts": "my-cluster-name-rs0"}
      2022-09-05T19:08:04.263Z        INFO    controller_psmdb        Cluster state changed   {"previous": "error", "current": "initializing"}
      2022-09-05T19:08:04.315Z        INFO    controller_psmdb        StatefulSet is changed, starting smart update   {"name": "my-cluster-name-rs0"}
      

      Some of the messages have namespace and some don't, it would be best for debugging reasons to have namespace in all if possible.

      Attachments

        Activity

          People

            Unassigned Unassigned
            tomislav.plavcic@percona.com Tomislav Plavcic
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Smart Checklist