Uploaded image for project: 'Percona XtraBackup'
  1. Percona XtraBackup
  2. PXB-1819

PXB 8.0 Freezes during prepare

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Priority: High
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 8.0.6
    • Component/s: None
    • Labels:
      None

      Description

      During the prepare phase, PXB freezes and never finish.  No IOPs, one CPU core at 100% or close.

       

      Been running for close to 4h+:

       

      xtrabackup --prepare --use-memory=100G --target-dir=. 2> xb.err | tee xb.out 
      

       

      CPU usage at 100%:

       

      [percona@s02 test_prepare]$ date; ps faxu | grep xtrabackup | grep -v grep
      Tue Mar 26 19:46:11 CET 2019
      percona 2952 99.9 12.8 114371772 16968048 pts/8 Rl+ 16:07 218:52 | \_ xtrabackup --prepare --use-memory=100G --target-dir=.
      [percona@s02 test_prepare]$ date; ps faxu | grep xtrabackup | grep -v grep
      Tue Mar 26 19:46:24 CET 2019
      percona 2952 99.9 12.8 114371772 16968048 pts/8 Rl+ 16:07 219:05 | \_ xtrabackup --prepare --use-memory=100G --target-dir=. 
      

       

      no IO:

      [percona@s02 test_prepare]$ date; cat /proc/2952/io
      Tue Mar 26 19:48:25 CET 2019
      rchar: 4372810104
      wchar: 573601435
      syscr: 71623
      syscw: 3552
      read_bytes: 93384704
      write_bytes: 573640704
      cancelled_write_bytes: 0
      [percona@s02 test_prepare]$ date; cat /proc/2952/io
      Tue Mar 26 19:48:41 CET 2019
      rchar: 4372810104
      wchar: 573601435
      syscr: 71623
      syscw: 3552
      read_bytes: 93384704
      write_bytes: 573640704
      cancelled_write_bytes: 0
      

      Looking at the pt-pmp output, the call stack is:

      sched_yield(libc.so.6),__gthread_yield(gthr-default.h:692),yield(gthr-default.h:692),Fil_shard::mutex_acquire_and_get_space(gthr-default.h:692),Fil_shard::space_extend(fil0fil.cc:6204),Fil_shard::get_file_for_io(fil0fil.cc:7311),Fil_shard::do_io(fil0fil.cc:7539),fil_io(fil0fil.cc:7812),buf_read_page_low(buf0rea.cc:138),buf_read_recv_pages(buf0rea.cc:786),recv_read_in_area(log0recv.cc:1124),recv_apply_log_rec(log0recv.cc:1124),recv_apply_hashed_log_recs(log0recv.cc:1124),srv_start(srv0start.cc:2439),innodb_init(xtrabackup.cc:2259),xtrabackup_prepare_func(xtrabackup.cc:6871),main(xtrabackup.cc:7817)

      All the other threads appear to be idle.  Attached is the pt-pmp output and the flamegraph (and perf data).

       

       

        Smart Checklist

          Attachments

          1. perf.report.out
            15 kB
          2. perf.script.out
            4.65 MB
          3. pt-pmp.out
            7 kB
          4. thread_apply_all_bt.txt
            166 kB
          5. xb_20190326.svg
            39 kB

            Activity

              People

              Assignee:
              sergei.glushchenko Sergei Glushchenko (Inactive)
              Reporter:
              yves@percona.com Yves Trudeau
              Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - Not Specified
                  Not Specified
                  Logged:
                  Time Spent - 1 day, 2 hours, 14 minutes
                  1d 2h 14m