Uploaded image for project: 'Percona XtraDB Cluster'
  1. Percona XtraDB Cluster
  2. PXC-2505

PXC-8.0 xtrabackup SST is failing with OS errno 32 - Broken pipe

Details

    • Admin & Maintenance Task
    • Status: Done
    • High
    • Resolution: Cannot Reproduce
    • 8.0.15-experimental
    • None
    • None

    Description

      PXC-8.0 xtrabackup SST is failing with OS errno 32 - Broken pipe. The node will be automatically up and running after a few such failures.

      Testcase

      1) Initialize PXC-5.7 data directories 
      
      ./pxc57/bin/mysqld --no-defaults --initialize-insecure --datadir=/dev/shm/qa/node1 --basedir=/dev/shm/qa/pxc57
      ./pxc57/bin/mysqld --no-defaults --initialize-insecure --datadir=/dev/shm/qa/node2 --basedir=/dev/shm/qa/pxc57
      
      2) start cluster 
      
      ./pxc57/bin/mysqld --defaults-file=/dev/shm/qa/pxc57_n1.cnf --wsrep-new-cluster &
      ./pxc57/bin/mysqld --defaults-file=/dev/shm/qa/pxc57_n2.cnf &
      
      3) Initiate sysbench load
      
      mysql  -A -uroot -S/dev/shm/qa/node1/socket.sock -e "create user [email protected]'%' identified with  mysql_native_password by 'test';"
      mysql  -A -uroot -S/dev/shm/qa/node1/socket.sock -e "grant all on *.* to [email protected]'%';"
      mysql  -A -uroot -S/dev/shm/qa/node1/socket.sock -e "drop database if exists sbtest;create database sbtest"
      
      
      sysbench /usr/share/sysbench/oltp_insert.lua --mysql-db=sbtest --mysql-user=sysbench --mysql-password=test --db-driver=mysql --mysql-socket=/dev/shm/qa/node1/socket.sock --threads=10 --tables=10 --table-size=1000 prepare
      
      sysbench /usr/share/sysbench/oltp_read_write.lua --mysql-db=sbtest --mysql-user=sysbench --mysql-password=test --db-driver=mysql --mysql-socket=/dev/shm/qa/node1/socket.sock --threads=10 --tables=10 --time=3000 --report-interval=10 run
      
      sleep 60
      
      4) shutdown node2 for upgrade
      
      /dev/shm/qa/pxc57/bin/mysqladmin -uroot -S/dev/shm/qa/node2/socket.sock shutdown
      
      sleep 150
      
      5) kill sysbench run
      
      # start node2 with PXC-8.0 binary
      
      ./pxc80/bin/mysqld --defaults-file=/dev/shm/qa/pxc80_n2.cnf &
      
      6) Check error log
      
      # If above steps are not reproduced the issue, please follow these steps too to reproduce the issue.
      
      1) shutdown cluster
      /dev/shm/qa/pxc80/bin/mysqladmin -uroot -S/dev/shm/qa/node2/socket.sock shutdown
      /dev/shm/qa/pxc80/bin/mysqladmin -uroot -S/dev/shm/qa/node1/socket.sock shutdown
      
      2) start node1 with PXC-8.0 binary
      
      ./pxc80/bin/mysqld --defaults-file=/dev/shm/qa/pxc80_n1.cnf --wsrep-new-cluster &
      
      3) initiate sysbench 
      
      sysbench /usr/share/sysbench/oltp_read_write.lua --mysql-db=sbtest --mysql-user=sysbench --mysql-password=test --db-driver=mysql --mysql-socket=/dev/shm/qa/node1/socket.sock --threads=10 --tables=10 --time=3000 --report-interval=10 run
      
      4) kill sysbench after 60 sec
      
      5) start node2
      
      ./pxc80/bin/mysqld --defaults-file=/dev/shm/qa/pxc80_n2.cnf &
      

      Attachments

        1. node1.err
          161 kB
        2. node2.err
          291 kB
        3. pxc57_n1.cnf
          0.9 kB
        4. pxc57_n2.cnf
          0.9 kB
        5. pxc80_n1.cnf
          0.9 kB
        6. pxc80_n2.cnf
          0.9 kB

        Issue Links

          Activity

            People

              Unassigned Unassigned
              ramesh.sivaraman Ramesh Sivaraman (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - Not Specified
                  Not Specified
                  Logged:
                  Time Spent - 2 hours
                  2h

                  Smart Checklist