Uploaded image for project: 'Percona Operator for MySQL based on Percona XtraDB Cluster'
  1. Percona Operator for MySQL based on Percona XtraDB Cluster
  2. K8SPXC-278

ProxySQL never getting ready status on some environments after the cluster launch

Details

    • Bug
    • Status: Done
    • Medium
    • Resolution: Fixed
    • 1.4.0
    • 1.6.0
    • None
    • None
    • Ubuntu v20.04

      K8s v1.17.5

      Rancher v2.4.2

    Description

      The proxysql-monit pod on each instance gets stuck in a crash loop indefinitely. I've spent two days trying to resolve it. Any ideas? I attached the log but here is the error:

       

      2020/05/07 20:16:15 Failed to execute /usr/bin/add_pxc_nodes.sh: + main 
      + echo 'Running /usr/bin/add_pxc_nodes.sh' 
      Running /usr/bin/add_pxc_nodes.sh 
      + read -ra first_host 
      + '[' -z db-cluster-1-pxc-0.db-cluster-1-pxc.pxc.svc.cluster.local ']' 
      ++ echo db-cluster-1-pxc-0.db-cluster-1-pxc.pxc.svc.cluster.local 
      ++ cut -d . -f 2- 
      + service=db-cluster-1-pxc.pxc.svc.cluster.local 
      + sleep 15s 
      + wait_for_mysql db-cluster-1-pxc.pxc.svc.cluster.local 
      + local h=db-cluster-1-pxc.pxc.svc.cluster.local 
      + echo 'Waiting for host db-cluster-1-pxc.pxc.svc.cluster.local to be online...' 
      Waiting for host db-cluster-1-pxc.pxc.svc.cluster.local to be online... 
      ++ mysql_root_exec db-cluster-1-pxc.pxc.svc.cluster.local 'select 1' 
      ++ local server=db-cluster-1-pxc.pxc.svc.cluster.local 
      ++ local 'query=select 1' 
      ++ MYSQL_PWD=1KO5UIduAdcQscyt 
      ++ timeout 600 mysql -h db-cluster-1-pxc.pxc.svc.cluster.local -uroot -s -NB -e 'select 1' 
      + '[' 1 '!=' 1 ']' 
      + wait_for_proxy 
      + local h=127.0.0.1 
      + echo 'Waiting for host 127.0.0.1 to be online...' 
      Waiting for host 127.0.0.1 to be online... 
      ++ proxysql_admin_exec 127.0.0.1 'select 1' 
      ++ local server=127.0.0.1 
      ++ local 'query=select 1' 
      ++ MYSQL_PWD=3BhMOqF5A521fy2Y 
      ++ timeout 600 mysql -h 127.0.0.1 -P6032 -u proxyadmin -s -NB -e 'select 1' 
      + '[' 1 '!=' 1 ']' 
      + SSL_ARG= 
      ++ proxysql_admin_exec 127.0.0.1 'SELECT variable_value FROM global_variables WHERE variable_name="mysql-have_ssl"' 
      ++ local server=127.0.0.1 
      ++ local 'query=SELECT variable_value FROM global_variables WHERE variable_name="mysql-have_ssl"' 
      ++ MYSQL_PWD=3BhMOqF5A521fy2Y 
      ++ timeout 600 mysql -h 127.0.0.1 -P6032 -u proxyadmin -s -NB -e 'SELECT variable_value FROM global_variables WHERE variable_name="mysql-have_ssl"' 
      + '[' true = true ']' 
      + SSL_ARG=--use-ssl=yes 
      + proxysql-admin --config-file=/etc/proxysql-admin.cnf --cluster-hostname=db-cluster-1-pxc-0.db-cluster-1-pxc.pxc.svc.cluster.local --enable --update-cluster --remove-all-servers --disable-updates --force --debug --use-ssl=yes 
      debug (line:2771) : Loading /etc/proxysql-admin.cnf 
      debug: ProxySQL datadir: /var/lib/proxysql 
      debug (line:712) : exec_sql : [email protected]:6032 ==> SELECT 1 
      mysql: [ERROR] Could not open required defaults file: /dev/stdin 
      mysql: [ERROR] Fatal error in defaults handling. Program aborted! 
      debug: --> query failed 1 
      ERROR (line:713) : ProxySQL connection check failed. 
      -- Could not connect to ProxySQL at localhost:6032 
      

      Attachments

        1. cluster.yml
          8 kB
        2. container.log
          4 kB
        3. proxysql.log
          7 kB

        Activity

          People

            tomislav.plavcic@percona.com Tomislav Plavcic
            lots0logs Dustin Falgout
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - Not Specified
                Not Specified
                Logged:
                Time Spent - 3 hours, 35 minutes
                3h 35m

                Smart Checklist