I had server running PMM set up with DHCP which was assigned different IP upon reboot. This breaks pmm-client (which is somewhat expected) but also makes restart to be very slow:
This is painful in shutdown updates etc. Perhaps some timeouts can be substantially lowered We also need to consider what is expected behavior in such case - we can have for example pmm-server on the start to re-negotiate its IP.
- Client restart should have a reasonable timeout value
- Client should compare what's in Consul vs current IP during check-network execution
- Client should automatically update Consul with new IP address
- Restart should either complete or generate error in under 10s
- Restart that times out should generate useful error message