Uploaded image for project: 'Percona XtraDB Cluster'
  1. Percona XtraDB Cluster
  2. PXC-1871

LP #1525300: Whole cluster freezes if one node goes full


    • Bug
    • Status: Done
    • Low
    • Resolution: Won't Fix
    • None
    • None
    • None


      **Reported in Launchpad by tobixen last update 19-10-2017 14:46:29

      $ mysqld --version
      mysqld Ver 5.6.26-74.0-56 for debian-linux-gnu on x86_64 (Percona XtraDB Cluster (GPL), Release rel74.0, Revision 624ef81, WSREP version 25.12, wsrep_25.12)

      Three-node setup.

      I trimmed down the disk space so the disk was almost full at one node, ran some big insert jobs at another node (one query insert into foo select * from bar ...). The node with the full disk did eventually die, and the cluster could live on with only two nodes active. This is good.

      Then I turned on log_slave_updates on the node with full disk, restarted percona and repeated the experiment, but now with many small inserts to make sure the binlog would grow. This caused the whole cluster to freeze ...

      (error log on node with full disk)
      2015-12-11 15:32:05 23823 [Warning] Disk is full writing '/var/lib/mysql/bin.000003' (Errcode: 28 - No space left on device). Waiting fo
      r someone to free space...
      2015-12-11 15:32:05 23823 [Warning] Retry in 60 secs. Message reprinted in 600 secs

      On the other nodes, all write queries got stuck in state "wsrep in pre-commit stage". (I didn't check read queries though).

      Please advice if you want more information.




            krunal.bauskar Krunal Bauskar (Inactive)
            lpjirasync lpjirasync (Inactive)
            0 Vote for this issue
            1 Start watching this issue



              Smart Checklist