Uploaded image for project: 'Percona Monitoring and Management'
  1. Percona Monitoring and Management
  2. PMM-2483

LVM in the PMM Server AMI is poorly configured/documented

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Priority: High
    • Resolution: Fixed
    • Affects Version/s: 1.9.1, 1.12.0
    • Fix Version/s: 1.16.0
    • Component/s: PMM Server
    • Labels:

      Description

      The PMM server AMI currently has a ticking time bomb built in, due to how it is configured. 

      The DataLV logical volume is currently built in a ThinPool LV. The ThinPool is made of a ThinPool_tdata and a ThinPool_tmeta volume. The default size of the ThinPool_tmeta LV, where metadata is stored, is only 16 megs (maybe because the default volume is only 8G?)

      Increasing the size of the DataLV/ThinPool does not increase the size of ThinPool_tmeta. Not being familiar with thin pools, I thought I had plenty of disk space after increasing the DataLV. I found pmm-server completely locked up today, only to finally realize that the metadata partition was full. (This lack of space does not show in df, only lvs reflects it)

      It's not clear to me what benefits the thin pool brings here. At a minimum, this feels like something that should be documented as part of the install process.

      The documentation here https://www.percona.com/doc/percona-monitoring-and-management/deploy/server/ami.html#resizing-the-ebs-volume mentions how to resize the ThinPool but does not mention anything about ensuring that the metadata portion is also large enough.

      As another option, it might be worth enabling the thin_pool_autoextend_threshold in /etc/lvm/lvm.conf?

      Here is the relevant section as it is in the AMI:

       

      # Configuration option activation/thin_pool_autoextend_threshold.
      # Auto-extend a thin pool when its usage exceeds this percent.
      # Setting this to 100 disables automatic extension.
      # The minimum value is 50 (a smaller value is treated as 50.)
      # Also see thin_pool_autoextend_percent.
      # Automatic extension requires dmeventd to be monitoring the LV.
      #
      # Example
      # Using 70% autoextend threshold and 20% autoextend size, when a 1G
      # thin pool exceeds 700M, it is extended to 1.2G, and when it exceeds
      # 840M, it is extended to 1.44G:
      # thin_pool_autoextend_threshold = 70
      #
      thin_pool_autoextend_threshold = 100
      

       Thank you 

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                olivier@ookla.com Olivier Mignault
              • Votes:
                4 Vote for this issue
                Watchers:
                13 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - Not Specified
                  Not Specified
                  Logged:
                  Time Spent - 1 day, 30 minutes
                  1d 30m