Uploaded image for project: 'Percona Monitoring and Management'
  1. Percona Monitoring and Management
  2. PMM-7310

VictoriaMetrics consuming 100's Gb's of disk in /tmp/searchResults in PMM 2.13.0

Details

    • Bug
    • Status: Done
    • Medium
    • Resolution: Fixed
    • 2.13.0
    • 2.14.0
    • VictoriaMetrics
    • None
    • Yes
    • [obsolete] C/S Core

    Description

      After upgrading from 2.11.1 to 2.13.0, VictoriaMetrics started to allocate a large number of files inside /srv/victoriametrics/data/tmp/searchResults and did not remove after use.

       

      This resulted in high consumption space, under a short period of time; Around 100GB per day.

       

      The current process to free up the data is to manually remove the files, followed by a restart in the container or in the VictoriaMetrics process, once even deleted the files, the file descriptors are kept open.

       

      PMM configuration:

      Approximately 200 instances with 2 or 3 services each

       

      What is expected:

      VM should remove temporary files that are not needed anymore

       

      Tested Scenario:

      Upgraded docker image from 2.11.1 to 2.13.0, created a concurrent execution by opening multiples dashboard requests, from to 2 monitored nodes, and 10 services each. Did not generate the search result files.

       

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              jeannunes.dasilva Jean Nunes da Silva
              Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - Not Specified
                  Not Specified
                  Logged:
                  Time Spent - 2 hours
                  2h

                  Smart Checklist