Uploaded image for project: 'Percona Monitoring and Management'
  1. Percona Monitoring and Management
  2. PMM-946

QAN sparklines drop to zero when data is not available

    Details

      Description

      Hi,

      I'm running tpcc benchmark and I can see the query rate is close to be sustaining:

       480, trx: 2182, 95%: 308.322, 99%: 596.208, max_rt: 793.474, 2170|792.662,217|791.926, 219|783.144, 220|916.316
       490, trx: 2128, 95%: 273.530, 99%: 611.023, max_rt: 890.880, 2134|914.858,213|609.174, 213|953.576, 211|1071.354
       500, trx: 1984, 95%: 366.341, 99%: 600.866, max_rt: 868.562, 1982|863.143,198|837.390, 198|904.934, 199|995.145
       510, trx: 1911, 95%: 445.560, 99%: 614.141, max_rt: 845.417, 1912|837.042,191|584.219, 190|857.055, 193|961.597
       520, trx: 2211, 95%: 274.268, 99%: 545.981, max_rt: 696.542, 2211|711.477,221|677.584, 221|718.634, 219|827.014
       530, trx: 1592, 95%: 508.589, 99%: 663.847, max_rt: 818.528, 1600|809.479,159|840.560, 159|851.124, 160|948.842
       540, trx: 2030, 95%: 384.774, 99%: 575.861, max_rt: 638.464, 2031|670.546,205|620.869, 204|653.726, 202|788.003
       550, trx: 1857, 95%: 459.925, 99%: 653.790, max_rt: 732.629, 1847|711.508,183|728.573, 186|789.136, 186|870.834
       560, trx: 2081, 95%: 381.334, 99%: 504.797, max_rt: 757.607, 2080|790.271,209|726.417, 208|784.793, 209|940.798
       570, trx: 1691, 95%: 489.764, 99%: 710.098, max_rt: 745.526, 1693|765.426,168|744.015, 168|863.856, 169|888.917
       580, trx: 1814, 95%: 486.549, 99%: 633.940, max_rt: 677.099, 1811|701.145,181|651.013, 183|730.814, 181|800.678
       590, trx: 2070, 95%: 297.266, 99%: 515.178, max_rt: 621.902, 2078|604.204,209|616.401, 207|646.966, 208|746.065
       600, trx: 1655, 95%: 561.226, 99%: 679.123, max_rt: 746.242, 1652|764.660,166|641.683, 166|770.223, 166|944.401
       610, trx: 1978, 95%: 391.396, 99%: 523.886, max_rt: 598.999, 1974|659.861,198|640.113, 197|648.080, 198|818.221
       620, trx: 1519, 95%: 529.880, 99%: 642.922, max_rt: 703.769, 1518|739.142,151|633.367, 152|716.827, 151|877.348
       630, trx: 1629, 95%: 548.931, 99%: 636.221, max_rt: 682.470, 1633|745.407,163|765.854, 162|726.662, 162|1410.887
       480, trx: 2182, 95%: 308.322, 99%: 596.208, max_rt: 793.474, 2170|792.662,217|791.926, 219|783.144, 220|916.316
       490, trx: 2128, 95%: 273.530, 99%: 611.023, max_rt: 890.880, 2134|914.858,213|609.174, 213|953.576, 211|1071.354
       500, trx: 1984, 95%: 366.341, 99%: 600.866, max_rt: 868.562, 1982|863.143,198|837.390, 198|904.934, 199|995.145
       510, trx: 1911, 95%: 445.560, 99%: 614.141, max_rt: 845.417, 1912|837.042,191|584.219, 190|857.055, 193|961.597
       520, trx: 2211, 95%: 274.268, 99%: 545.981, max_rt: 696.542, 2211|711.477,221|677.584, 221|718.634, 219|827.014
       530, trx: 1592, 95%: 508.589, 99%: 663.847, max_rt: 818.528, 1600|809.479,159|840.560, 159|851.124, 160|948.842
       540, trx: 2030, 95%: 384.774, 99%: 575.861, max_rt: 638.464, 2031|670.546,205|620.869, 204|653.726, 202|788.003
       550, trx: 1857, 95%: 459.925, 99%: 653.790, max_rt: 732.629, 1847|711.508,183|728.573, 186|789.136, 186|870.834
       560, trx: 2081, 95%: 381.334, 99%: 504.797, max_rt: 757.607, 2080|790.271,209|726.417, 208|784.793, 209|940.798
       570, trx: 1691, 95%: 489.764, 99%: 710.098, max_rt: 745.526, 1693|765.426,168|744.015, 168|863.856, 169|888.917
       580, trx: 1814, 95%: 486.549, 99%: 633.940, max_rt: 677.099, 1811|701.145,181|651.013, 183|730.814, 181|800.678
       590, trx: 2070, 95%: 297.266, 99%: 515.178, max_rt: 621.902, 2078|604.204,209|616.401, 207|646.966, 208|746.065
       600, trx: 1655, 95%: 561.226, 99%: 679.123, max_rt: 746.242, 1652|764.660,166|641.683, 166|770.223, 166|944.401
       610, trx: 1978, 95%: 391.396, 99%: 523.886, max_rt: 598.999, 1974|659.861,198|640.113, 197|648.080, 198|818.221
       620, trx: 1519, 95%: 529.880, 99%: 642.922, max_rt: 703.769, 1518|739.142,151|633.367, 152|716.827, 151|877.348
       630, trx: 1629, 95%: 548.931, 99%: 636.221, max_rt: 682.470, 1633|745.407,163|765.854, 162|726.662, 162|1410.887
      

      Yet QAN2 shows

      [image: Inline image 1]

      Which shows as if QPS has dropped for last inverval(s)

      It is interesting though QAN1 shows different graph in Count:

      [image: Inline image 2]

      Why are they different ?

      It does not show it as correct ether but it does not have such long gap

      Potential Solution

      QAN data comes in with 1-2 min delay (typical) though it is possible to be delayed more with bad network and for example when MySQL is data

      QAN need to be able to understand what "no data" is different then 0 - as such I would expect the Sparkline to terminate earlier than the whole interval but end on the high value (not drop down) indicating the number of queries for last few min is not available yet rather than what it is 0

       

      TODO: detect and honor in Sparkline Intervals where we don't have complete data and not "drop them to zero" 

        Smart Checklist

          Attachments

          1. image.png
            image.png
            16 kB
          2. image.png
            image.png
            7 kB
          3. qan_1hr_drop_to_zero.PNG
            qan_1hr_drop_to_zero.PNG
            6 kB
          4. qan_5min_drop_to_zero.PNG
            qan_5min_drop_to_zero.PNG
            5 kB
          5. qan_1hr_mid_range_247.PNG
            qan_1hr_mid_range_247.PNG
            5 kB
          6. Screen Shot 2018-07-02 at 5.12.10 PM.png
            Screen Shot 2018-07-02 at 5.12.10 PM.png
            327 kB
          7. pmm-946_no_drop_when_series_is_zero.PNG
            pmm-946_no_drop_when_series_is_zero.PNG
            4 kB
          8. pmm-946_no_drop_when_series_is_zero_B.PNG
            pmm-946_no_drop_when_series_is_zero_B.PNG
            3 kB

            Issue Links

              Activity

                People

                • Assignee:
                  Unassigned
                  Reporter:
                  pz Peter Zaitsev
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  4 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Time Tracking

                    Estimated:
                    Original Estimate - Not Specified
                    Not Specified
                    Remaining:
                    Remaining Estimate - 0 minutes
                    0m
                    Logged:
                    Time Spent - 1 day, 3 hours, 30 minutes
                    1d 3h 30m