Old data (from Prometheus) not deleted when Retention period expires

Impact on the user:

• Storage on the PMM server can be full and this will lead PMM to not work at all.

STR:

1. Setup pmm 2.11.1 with default 30 days data retention period
2. Wait for data retention period + days
3. Update to PMM 2.12.0
4. Wait for 1 day 

Expected Result:

• The data for the day -31 and older -  will be deleted

Given Result:

• Old Data still exists and consume storage 

How to test:

• Setup pmm 2.11.1 
• Change data retention period
• Wait for data retention period + days
• Update to PMM 2.12.0
• Wait for 1 day 

Workaround:
N/A

Original report:

On PMM demo (Retention 30 days, updated  to 2.12.0 at 9 Dec, 2020) we can still see data for Nov 1. (This is 45 days already passed)

https://pmmdemo.percona.com/graph/d/prometheus-advanced/advanced-data-exploration?orgId=1&refresh=1m&from=now-50d&to=now