Uploaded image for project: 'Percona Monitoring and Management'
  1. Percona Monitoring and Management
  2. PMM-9862

Bogus "Exporter is not Connected" for MongoDB ReplicaSet status

Details

    • Bug
    • Status: Done
    • Medium
    • Resolution: Cannot Reproduce
    • 2.26.0
    • None
    • None
    • 1
    • Yes
    • Yes
    • Yes
    • C/S Core

    Description

      PMM periodically shows the bogus status "Exporter is not Connected" at the "ReplSet States" graph on the "MongoDB ReplSet Summary" dashboard.

      How to Repeat

      1. Setup 3-nodes MongoDB replica set, for example, with mlaunch
      2. Install PMM
      3. Setup agents. At this step it does not matter if they can connect to MongoDB instances or not. They just need to be added to PMM.
        sudo pmm-admin add mongodb --replication-set=myset --port=27017 127.0.0.1:27017 --service-name=automongo --metrics-mode=auto
        
        sudo pmm-admin add mongodb --replication-set=myset --port=27018 127.0.0.1:27018 --service-name=pushmongo --metrics-mode=push
        
        sudo pmm-admin add mongodb --replication-set=myset --port=27019 127.0.0.1:27019 --service-name=pullmongo --metrics-mode=pull
        
      4. Wait 5 minutes or more, then remove these agents:
        sudo pmm-admin remove mongodb pullmongo
        sudo pmm-admin remove mongodb pushmongo
        sudo pmm-admin remove mongodb automongo
        
      5. Add agents with the same names.
        sudo pmm-admin add mongodb --replication-set=myset --port=27017 127.0.0.1:27017 --service-name=automongo --metrics-mode=auto
        
        sudo pmm-admin add mongodb --replication-set=myset --port=27018 127.0.0.1:27018 --service-name=pushmongo --metrics-mode=push
        
        sudo pmm-admin add mongodb --replication-set=myset --port=27019 127.0.0.1:27019 --service-name=pullmongo --metrics-mode=pull
        
      6. Now ensure that agents can connect to MongoDB and PMM shows status for them on the "MongoDB ReplSet Summary" dashboard.
      7. So far so good.
      8. Wait for 12 hours, so agents that you removed do not show up for the "Last 12 hours" interval. The bug can be seen in the "Last 6 hours" and "Last 3 hours" intervals too but larger the interval you take, more chances to see the issue you have.
      9. Now is the tricky part: replicas need to change PRIMARY. I tried to force this operation manually, but the most reliable way to repeat the issue is to put laptop to the sleep mode, then wake it up. Then eventually you will see the picture as in the attached screenshot bogus_status.png: pushmongo is PRIMARY, pullmongo is SECONDARY, and automongo is in the bogus status "Exporter is not Connected".
      10. Now check data for the server that is in the bogus status, wait for 5 minutes (for 12 hours interval) or 1 minute (for 3 hours interval as in my example) and check data again. Sort data by time descending. You will see that it does not have data only for the upper row, no matter which time it is now. Screenshot 20-28-18.png, then 20-29-20.png
      11. Now go to the smaller interval, say, 15 minutes and see that it shows the status of the same server properly. Data for the last moment is also present. Screenshots last15.png and last15_data.png
      12. So I would say this is not exporter issue, rather how PMM uses low and medium resolution metrics

      Attachments

        1. 20-28-18.png
          20-28-18.png
          87 kB
        2. 20-29-20.png
          20-29-20.png
          83 kB
        3. bogus_status.png
          bogus_status.png
          289 kB
        4. last15_data.png
          last15_data.png
          101 kB
        5. last15.png
          last15.png
          240 kB

        Issue Links

          Activity

            People

              anton.bystrov Anton Bystrov
              sveta.smirnova Sveta Smirnova
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Smart Checklist