Affects Version/s: None
Fix Version/s: None
Red Hat Enterprise Linux 7.2
I am using proxysql (version 1.3.7) as a load balancer for my MariaDB Galera cluster (version 10.1.25). All the servers are running on RHEL 7.
Yesterday one of the servers running ProxySQL started to have a high CPU load.
When i checked to see the source of the problem, i found the following things:
- As i ran ps, i noticed over 50 processes belonging to proxysql running the following command (with the number of processes increasing every few seconds):
- After trying different things i found the problem was that when trying to connect to two of my Galera nodes, the mysql command hangs indefinitely (The mysqld on those servers was in an faulty state - mysqld was listening on standard port, but would not respond) .
- I tried adding 'timeout' before the mysql command in the mysql_exec function in proxysql_node_monitor (as it is done in proxysql_galera_checker) -
- I also added the TIMEOUT parameter to the script, and set its value in both proxysql_node_monitor and proxysql_galera_checker (where it already existed) to 1.
After taking the those actions, ProxySQL returned to normal, marking the non responding Galera nodes as 'OFFLINE_HARD' and directing traffic to the operational nodes.
When i checked the latest version of proxysql_node_monitor, i saw that mysql_exec still has no timeout definition, so i think this issue is relevant for the latest version too.