Enormous space usage and slow queries for Group replication monitoring

 
Impact on the user:

• PMM (VM) storage is growing fast and consumes a lot of space, and performs slowly if the instance has Group Replication monitored 

Steps to reproduce:

1. Install PMM
2. Add instances with Group Replication
3. let pmm monitor it fr some time 

Actual result:

• significant increase of used space for pmm server 
• Increased cardinality of data in PMM

Expected Result:

• usual increase for added instances on pmm server  

Workaround:
Remove lines  https://github.com/percona/mysqld_exporter/blob/45c70da42accde2b462b376b98a0ea902a09cd0e/queries-mysqld-group-replication.yml#L121-L126  or remove queries-mysqld-group-replication.yml file from client-side completely.

This will disable some data causing the problem or full group replication monitoring. 

Details

The problem is caused by High-cardinality data generated by Custom queries for Group replication monitoring.

Especially lines https://github.com/percona/mysqld_exporter/blob/45c70da42accde2b462b376b98a0ea902a09cd0e/queries-mysqld-group-replication.yml#L121-L126 

They are exposing labels = "a75335af-b706-49af-a99d-53e8a2004fcb:4481072264" and the values are unique every time and this causing creation a lot of time series on the system. 

Proposed solution:

1. Remove text labels causing hight-cardinality in GR SQL
2. Remove Transaction Details table from GR  dashboard
3. Improve  GR transaction  data later in the scope of PMM-9980
4. Provide users with instructions on how to clean up old Time-series causing performance problems

(^ removal and improvement later confirmed with [email protected] )