Uploaded image for project: 'ProxySQL Admin Scripts'
  1. ProxySQL Admin Scripts
  2. PSQLADM-8

ProxySQL High CPU load when mysqld hangs

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Priority: Medium
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Environment:

      Red Hat Enterprise Linux 7.2

      ProxySQL 1.3.7

      MariaDB 10.1.25

      Description

      Hello

      I am using proxysql (version 1.3.7) as a load balancer for my MariaDB Galera cluster (version 10.1.25). All the servers are running on RHEL 7.

      Yesterday one of the servers running ProxySQL started to have a high CPU load.

      When i checked to see the source of the problem, i found the following things:

      • As i ran ps, i noticed over 50 processes belonging to proxysql running the following command (with the number of processes increasing every few seconds):
        • mysql --defaults-file=/dev/stdin --protocol=tcp -nNE "SHOW STATUS LIKE 'wsrep_incoming_addresses'"
      • After trying different things i found the problem was that when trying to connect to two of my Galera nodes, the mysql command hangs indefinitely (The mysqld on those servers was in an faulty state - mysqld was listening on standard port, but would not respond) .
      • I tried adding 'timeout' before the mysql command in the mysql_exec function in proxysql_node_monitor (as it is done in proxysql_galera_checker) -
        • timeout $TIMEOUT mysql --defaults-file=/dev/stdin --protocol=tcp -Bse "${query}" 2>/dev/null
      • I also added the TIMEOUT parameter to the script, and set its value in both proxysql_node_monitor and proxysql_galera_checker (where it already existed) to 1.

      After taking the those actions, ProxySQL returned to normal, marking the non responding Galera nodes as 'OFFLINE_HARD' and directing traffic to the operational nodes.

       

      When i checked the latest version of proxysql_node_monitor, i saw that mysql_exec still has no timeout definition, so i think this issue is relevant for the latest version too.

        Smart Checklist

          Attachments

            Activity

              People

              • Assignee:
                ramesh.sivaraman Ramesh Sivaraman
                Reporter:
                nitai-nahaliel nitai nahaliel
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: