Uploaded image for project: 'Percona XtraDB Cluster'
  1. Percona XtraDB Cluster
  2. PXC-2285

A lot of WSREP: Failed to report last committed <number>, -110 (Connection timed out)

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Priority: Medium
    • Resolution: Incomplete
    • Affects Version/s: 5.6.39-26.25
    • Fix Version/s: None
    • Component/s: None

      Description

      I'm using  Percona-XtraDB-Cluster-server-56 with Version : 5.6.39 Release : 26.25.1.el6.

      with 2 node + 1 percona-xtradb-cluster-garbd.

      I'm facing following problem. PXC log report a a lot of warning 

      2018-11-19 17:33:48 19414 [Warning] WSREP: Failed to report last committed 1327742612, -110 (Connection timed out)
      2018-11-19 19:07:06 19414 [Warning] WSREP: Failed to report last committed 1327963155, -110 (Connection timed out)
      2018-11-19 19:26:58 19414 [Warning] WSREP: Failed to report last committed 1328000375, -110 (Connection timed out)
      2018-11-19 21:11:49 19414 [Warning] WSREP: Failed to report last committed 1328209550, -110 (Connection timed out)
      2018-11-19 23:44:14 19414 [Warning] WSREP: Failed to report last committed 1328381758, -110 (Connection timed out)
      2018-11-20 09:31:48 19414 [Warning] WSREP: Failed to report last committed 1328978200, -110 (Connection timed out)
      2018-11-20 09:43:42 19414 [Warning] WSREP: Failed to report last committed 1329040974, -110 (Connection timed out)
      2018-11-20 09:46:06 19414 [Warning] WSREP: Failed to report last committed 1329052427, -110 (Connection timed out)
      2018-11-20 10:20:36 19414 [Warning] WSREP: Failed to report last committed 1329227873, -110 (Connection timed out)
      2018-11-20 10:28:20 19414 [Warning] WSREP: Failed to report last committed 1329265424, -110 (Connection timed out)
      2018-11-20 11:03:22 19414 [Warning] WSREP: Failed to report last committed 1329453317, -110 (Connection timed out)
      2018-11-20 11:23:47 19414 [Warning] WSREP: Failed to report last committed 1329559539, -110 (Connection timed out)
      2018-11-20 11:25:00 19414 [Warning] WSREP: Failed to report last committed 1329564657, -110 (Connection timed out)
      2018-11-20 11:33:53 19414 [Warning] WSREP: Failed to report last committed 1329604492, -110 (Connection timed out)
      2018-11-20 11:57:34 19414 [Warning] WSREP: Failed to report last committed 1329715728, -110 (Connection timed out)
      2018-11-20 13:42:23 19414 [Warning] WSREP: Failed to report last committed 1330147628, -110 (Connection timed out)
      2018-11-20 13:43:00 19414 [Warning] WSREP: Failed to report last committed 1330149773, -110 (Connection timed out)
      2018-11-20 14:27:01 19414 [Warning] WSREP: Failed to report last committed 1330379363, -110 (Connection timed out)
      2018-11-20 14:30:12 19414 [Warning] WSREP: Failed to report last committed 1330391879, -110 (Connection timed out)
      2018-11-20 14:32:14 19414 [Warning] WSREP: Failed to report last committed 1330401771, -110 (Connection timed out)
      2018-11-20 14:34:41 19414 [Warning] WSREP: Failed to report last committed 1330411552, -110 (Connection timed out)
      2018-11-20 14:35:15 19414 [Warning] WSREP: Failed to report last committed 1330413554, -110 (Connection timed out)
      2018-11-20 14:36:37 19414 [Warning] WSREP: Failed to report last committed 1330419814, -110 (Connection timed out)
      2018-11-20 14:37:26 19414 [Warning] WSREP: Failed to report last committed 1330422477, -110 (Connection timed out)
      2018-11-20 14:41:02 19414 [Warning] WSREP: Failed to report last committed 1330437542, -110 (Connection timed out)
      2018-11-20 14:50:49 19414 [Warning] WSREP: Failed to report last committed 1330484259, -110 (Connection timed out)
      2018-11-20 14:57:32 19414 [Warning] WSREP: Failed to report last committed 1330517211, -110 (Connection timed out)
      2018-11-20 15:01:34 19414 [Warning] WSREP: Failed to report last committed 1330536212, -110 (Connection timed out)
      2018-11-20 15:43:19 19414 [Warning] WSREP: Failed to report last committed 1330769642, -110 (Connection timed out)
      2018-11-20 15:50:59 19414 [Warning] WSREP: Failed to report last committed 1330808893, -110 (Connection timed out)
      2018-11-20 15:59:07 19414 [Warning] WSREP: Failed to report last committed 1330848316, -110 (Connection timed out)
      2018-11-20 16:08:41 19414 [Warning] WSREP: Failed to report last committed 1330891310, -110 (Connection timed out)
      2018-11-20 16:10:22 19414 [Warning] WSREP: Failed to report last committed 1330898081, -110 (Connection timed out)
      2018-11-20 16:13:43 19414 [Warning] WSREP: Failed to report last committed 1330913054, -110 (Connection timed out)
      2018-11-20 16:19:46 19414 [Warning] WSREP: Failed to report last committed 1330940790, -110 (Connection timed out)
      2018-11-20 16:23:10 19414 [Warning] WSREP: Failed to report last committed 1330954782, -110 (Connection timed out)
      2018-11-20 16:23:53 19414 [Warning] WSREP: Failed to report last committed 1330957407, -110 (Connection timed out)
      2018-11-20 16:26:01 19414 [Warning] WSREP: Failed to report last committed 1330967791, -110 (Connection timed out)
      2018-11-20 16:29:57 19414 [Warning] WSREP: Failed to report last committed 1330984126, -110 (Connection timed out)
      2018-11-20 16:32:22 19414 [Warning] WSREP: Failed to report last committed 1330994398, -110 (Connection timed out)
      2018-11-20 16:40:44 19414 [Warning] WSREP: Failed to report last committed 1331034549, -110 (Connection timed out)
      2018-11-20 16:49:01 19414 [Warning] WSREP: Failed to report last committed 1331068494, -110 (Connection timed out)
      2018-11-20 17:09:53 19414 [Warning] WSREP: Failed to report last committed 1331161202, -110 (Connection timed out)
      2018-11-20 17:10:53 19414 [Warning] WSREP: Failed to report last committed 1331164772, -110 (Connection timed out)

       

      It make any SQL transaction like insert and update in commit stage very slow, usual take 10-11 sec to finish. Normaly that sql only take less then 1 second.

      The ping between 2 node PXC is less than 0.2ms, and no drop package.

      Following is my.conf. similar on 2 node.

       

      [client]
      default-character-set = utf8mb4
      [mysql]
      default-character-set = utf8mb4
      # CLIENT #
      port = 3306
      socket = /u01/mysql-data/mysql/mysql.sock
      [mysqld]
      character-set-client-handshake = TRUE
      init-connect='SET NAMES utf8mb4'
      character-set-server = utf8mb4
      collation-server = utf8mb4_unicode_ci 
      lower_case_table_names=1
      # Disabling symbolic-links is recommended to prevent assorted security risks
      symbolic-links=0
      
      # Path to Galera library
      wsrep_provider=/usr/lib64/libgalera_smm.so
      
      # Cluster connection URL contains IPs of node#1, node#2
      wsrep_cluster_address=gcomm://192.168.5.50,192.168.5.123,192.168.5.124
      # In order for Galera to work correctly binlog format should be ROW
      binlog_format=ROW
      # MyISAM storage engine has only experimental support
      default_storage_engine=InnoDB
      # This changes how InnoDB autoincrement locks are managed and is a requirement for Galera
      innodb_autoinc_lock_mode=2
      # Node #6 address
      wsrep_node_address=192.168.5.124
      # Cluster name
      wsrep_cluster_name=my_centos_cluster
      # SST method
      wsrep_sst_method=xtrabackup-v2
      #Authentication for SST method
      wsrep_sst_auth="sstuser:3uro6OraYIT3"
      # GENERAL #
      user = mysql
      default-storage-engine = InnoDB
      socket = /u01/mysql-data/mysql/mysql.sock
      pid-file = /u01/mysql-data/mysql/mysql.pid
      # MyISAM #
      key-buffer-size = 32M
      myisam-recover-options = FORCE,BACKUP
      # SAFETY #
      max-allowed-packet = 16M
      max-connect-errors = 1000000
      sql-mode = ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_AUTO_VALUE_ON_ZERO,NO_ENGINE_SUBSTITUTION,NO_ZERO_DATE,NO_ZERO_IN_DATE
      # DATA STORAGE #
      datadir = /u01/mysql-data/mysql/
      # BINARY LOGGING #
      log-bin = /u01/mysql-data/mysql/mysql-bin
      expire-logs-days = 14
      sync-binlog = 1
      # CACHES AND LIMITS #
      tmp-table-size = 32M
      max-heap-table-size = 32M
      query-cache-type = 0
      query-cache-size = 0
      max-connections = 1000
      thread-cache-size = 200
      open-files-limit = 65535
      table-definition-cache = 4096
      table-open-cache = 4096
      # INNODB #
      innodb-flush-method = O_DIRECT
      innodb-log-files-in-group = 2
      innodb-log-file-size = 512M
      innodb-flush-log-at-trx-commit = 1
      innodb-file-per-table = 1
      innodb-buffer-pool-size = 42G
      # LOGGING #
      log-error = /u01/mysql-data/log/mysql-error.log
      log-queries-not-using-indexes = 1
      slow-query-log = 0
      slow-query-log-file = /u01/mysql-data/log/mysql-slow.log
      tmpdir = /u01/mysql-data/tmp-dir
      

       

       

       

       

       

       

        Smart Checklist

          Attachments

            Activity

              People

              Assignee:
              przemyslaw.malkowski@percona.com Przemyslaw Malkowski
              Reporter:
              lamda0511 Lam Viet
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: