-
Type:
Bug
-
Status: Done
-
Priority:
High
-
Resolution: Fixed
-
Affects Version/s: None
-
Fix Version/s: 8.0.6
-
Component/s: None
-
Labels:None
During the prepare phase, PXB freezes and never finish. No IOPs, one CPU core at 100% or close.
Been running for close to 4h+:
xtrabackup --prepare --use-memory=100G --target-dir=. 2> xb.err | tee xb.out
CPU usage at 100%:
[percona@s02 test_prepare]$ date; ps faxu | grep xtrabackup | grep -v grep Tue Mar 26 19:46:11 CET 2019 percona 2952 99.9 12.8 114371772 16968048 pts/8 Rl+ 16:07 218:52 | \_ xtrabackup --prepare --use-memory=100G --target-dir=. [percona@s02 test_prepare]$ date; ps faxu | grep xtrabackup | grep -v grep Tue Mar 26 19:46:24 CET 2019 percona 2952 99.9 12.8 114371772 16968048 pts/8 Rl+ 16:07 219:05 | \_ xtrabackup --prepare --use-memory=100G --target-dir=.
no IO:
[percona@s02 test_prepare]$ date; cat /proc/2952/io Tue Mar 26 19:48:25 CET 2019 rchar: 4372810104 wchar: 573601435 syscr: 71623 syscw: 3552 read_bytes: 93384704 write_bytes: 573640704 cancelled_write_bytes: 0 [percona@s02 test_prepare]$ date; cat /proc/2952/io Tue Mar 26 19:48:41 CET 2019 rchar: 4372810104 wchar: 573601435 syscr: 71623 syscw: 3552 read_bytes: 93384704 write_bytes: 573640704 cancelled_write_bytes: 0
Looking at the pt-pmp output, the call stack is:
sched_yield(libc.so.6),__gthread_yield(gthr-default.h:692),yield(gthr-default.h:692),Fil_shard::mutex_acquire_and_get_space(gthr-default.h:692),Fil_shard::space_extend(fil0fil.cc:6204),Fil_shard::get_file_for_io(fil0fil.cc:7311),Fil_shard::do_io(fil0fil.cc:7539),fil_io(fil0fil.cc:7812),buf_read_page_low(buf0rea.cc:138),buf_read_recv_pages(buf0rea.cc:786),recv_read_in_area(log0recv.cc:1124),recv_apply_log_rec(log0recv.cc:1124),recv_apply_hashed_log_recs(log0recv.cc:1124),srv_start(srv0start.cc:2439),innodb_init(xtrabackup.cc:2259),xtrabackup_prepare_func(xtrabackup.cc:6871),main(xtrabackup.cc:7817)
All the other threads appear to be idle. Attached is the pt-pmp output and the flamegraph (and perf data).
