Uploaded image for project: 'Percona Server for MySQL'
  1. Percona Server for MySQL
  2. PS-311

On server shutdown threadpool is crashing in debug mode due to race condition

    Details

    • Type: Bug
    • Status: Done
    • Priority: High
    • Resolution: Fixed
    • Affects Version/s: 5.7.x
    • Fix Version/s: None
    • Component/s: None

      Description

      There is a race condition that causes debug server to crash. The race is between main server thread and main worker thread. The stack-trace:

      15:04:24 Program terminated with signal SIGABRT, Aborted.
       15:04:24 #0  0x00007f9341966c37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
       15:04:24 56    ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
       15:04:24 #0  0x00007f9341966c37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
       15:04:24 #1  0x00007f934196a028 in __GI_abort () at abort.c:89
       15:04:24 #2  0x00007f934195fbf6 in _assert_fail_base (fmt=0x7f9341ab4018 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x1e24536 "THR_THD_initialized", file=file@entry=0x1e244e0 "/mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/sql/mysqld.h", line=line@entry=392, function=function@entry=0x1e24fa0 <_ZZL21my_thread_get_THR_THDvE19PRETTY_FUNCTION_> "THD* my_thread_get_THR_THD()") at assert.c:92
       15:04:24 #3  0x00007f934195fca2 in _GI_assert_fail (assertion=0x1e24536 "THR_THD_initialized", file=0x1e244e0 "/mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/sql/mysqld.h", line=392, function=0x1e24fa0 <_ZZL21my_thread_get_THR_THDvE19PRETTY_FUNCTION_> "THD* my_thread_get_THR_THD()") at assert.c:101
       15:04:24 #4  0x0000000000e8edfc in my_thread_get_THR_THD () at /mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/sql/mysqld.h:392
       15:04:24 #5  0x0000000000e8f1b6 in handle_fatal_signal (sig=6) at /mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/sql/signal_handler.cc:140
       15:04:24 #6  <signal handler called>
       15:04:24 #7  0x00007f9341966c37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
       15:04:24 #8  0x00007f934196a028 in __GI_abort () at abort.c:89
       15:04:24 #9  0x00007f934195fbf6 in _assert_fail_base (fmt=0x7f9341ab4018 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x2214b8d "THR_KEY_mysys_initialized", file=file@entry=0x2214b30 "/mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/mysys/my_thr_init.c", line=line@entry=75, function=function@entry=0x2214c80 <PRETTY_FUNCTION_.11273> "set_mysys_thread_var") at assert.c:92
       15:04:24 #10 0x00007f934195fca2 in _GI_assert_fail (assertion=0x2214b8d "THR_KEY_mysys_initialized", file=0x2214b30 "/mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/mysys/my_thr_init.c", line=75, function=0x2214c80 <PRETTY_FUNCTION_.11273> "set_mysys_thread_var") at assert.c:101
       15:04:24 #11 0x000000000189c25d in set_mysys_thread_var (mysys_var=0x0) at /mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/mysys/my_thr_init.c:75
       15:04:24 #12 0x000000000189cb90 in my_thread_end () at /mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/mysys/my_thr_init.c:370
       15:04:24 #13 0x0000000001673f07 in worker_main (param=0x2d61a00 <all_groups>) at /mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/sql/threadpool_unix.cc:1651
       15:04:24 #14 0x0000000001ce9361 in pfs_spawn_thread (arg=0x42e34a0) at /mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/storage/perfschema/pfs.cc:2190
       15:04:24 #15 0x00007f9342531184 in start_thread (arg=0x7f9333fbe700) at pthread_create.c:312
       15:04:24 #16 0x00007f9341a2dffd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
       15:04:24 
       15:04:24 Thread 2 (Thread 0x7f9343dfd780 (LWP 24033)):
       15:04:24 #0  os_event::~os_event (this=0x190fed0 <GenericPolicy<TTASEventMutex<GenericPolicy> >::~GenericPolicy()+42>, __in_chrg=<optimized out>) at /mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/storage/innobase/os/os0event.cc:238
       15:04:24 #1  0x000000000190d338 in TTASEventMutex<GenericPolicy>::~TTASEventMutex (this=0x2d7bd20 <ibuf_bitmap_mutex>, __in_chrg=<optimized out>) at /mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/storage/innobase/include/ib0mutex.h:627
       15:04:24 #2  0x000000000190f5a2 in PolicyMutex<TTASEventMutex<GenericPolicy> >::~PolicyMutex (this=0x2d7bd20 <ibuf_bitmap_mutex>, __in_chrg=<optimized out>) at /mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/storage/innobase/include/ib0mutex.h:926
       15:04:24 #3  0x00007f934196c1a9 in _run_exit_handlers (status=0, listp=0x7f9341cf26c8 <_exit_funcs>, run_list_atexit=run_list_atexit@entry=true) at exit.c:82
       15:04:24 #4  0x00007f934196c1f5 in __GI_exit (status=<optimized out>) at exit.c:104
       15:04:24 #5  0x0000000000e77e99 in mysqld_exit (exit_code=0) at /mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/sql/mysqld.cc:1206
       15:04:24 #6  0x0000000000e80060 in mysqld_main (argc=94, argv=0x392bd58) at /mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/sql/mysqld.cc:5445
       15:04:24 #7  0x0000000000e7637d in main (argc=9, argv=0x7fffe391c5b8) at /mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/sql/main.cc:25
       15:04:24 
       15:04:24 Thread 1 (Thread 0x7f9333fbe700 (LWP 24116)):
       15:04:24 #0  0x00007f9341966c37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
       15:04:24 #1  0x00007f934196a028 in __GI_abort () at abort.c:89
       15:04:24 #2  0x00007f934195fbf6 in _assert_fail_base (fmt=0x7f9341ab4018 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x1e24536 "THR_THD_initialized", file=file@entry=0x1e244e0 "/mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/sql/mysqld.h", line=line@entry=392, function=function@entry=0x1e24fa0 <_ZZL21my_thread_get_THR_THDvE19PRETTY_FUNCTION_> "THD* my_thread_get_THR_THD()") at assert.c:92
       15:04:24 #3  0x00007f934195fca2 in _GI_assert_fail (assertion=0x1e24536 "THR_THD_initialized", file=0x1e244e0 "/mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/sql/mysqld.h", line=392, function=0x1e24fa0 <_ZZL21my_thread_get_THR_THDvE19PRETTY_FUNCTION_> "THD* my_thread_get_THR_THD()") at assert.c:101
       15:04:24 #4  0x0000000000e8edfc in my_thread_get_THR_THD () at /mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/sql/mysqld.h:392
       15:04:24 #5  0x0000000000e8f1b6 in handle_fatal_signal (sig=6) at /mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/sql/signal_handler.cc:140
       15:04:24 #6  <signal handler called>
       15:04:24 #7  0x00007f9341966c37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
       15:04:24 #8  0x00007f934196a028 in __GI_abort () at abort.c:89
       15:04:24 #9  0x00007f934195fbf6 in _assert_fail_base (fmt=0x7f9341ab4018 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x2214b8d "THR_KEY_mysys_initialized", file=file@entry=0x2214b30 "/mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/mysys/my_thr_init.c", line=line@entry=75, function=function@entry=0x2214c80 <PRETTY_FUNCTION_.11273> "set_mysys_thread_var") at assert.c:92
       15:04:24 #10 0x00007f934195fca2 in _GI_assert_fail (assertion=0x2214b8d "THR_KEY_mysys_initialized", file=0x2214b30 "/mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/mysys/my_thr_init.c", line=75, function=0x2214c80 <PRETTY_FUNCTION_.11273> "set_mysys_thread_var") at assert.c:101
       15:04:24 #11 0x000000000189c25d in set_mysys_thread_var (mysys_var=0x0) at /mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/mysys/my_thr_init.c:75
       15:04:24 #12 0x000000000189cb90 in my_thread_end () at /mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/mysys/my_thr_init.c:370
       15:04:24 #13 0x0000000001673f07 in worker_main (param=0x2d61a00 <all_groups>) at /mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/sql/threadpool_unix.cc:1651
       15:04:24 #14 0x0000000001ce9361 in pfs_spawn_thread (arg=0x42e34a0) at /mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/storage/perfschema/pfs.cc:2190
       15:04:24 #15 0x00007f9342531184 in start_thread (arg=0x7f9333fbe700) at pthread_create.c:312
       15:04:24 #16 0x00007f9341a2dffd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

        Smart Checklist

          Attachments

            Activity

              People

              • Assignee:
                robert.golebiowski Robert Golebiowski
                Reporter:
                robert.golebiowski Robert Golebiowski
              • Votes:
                1 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 1 hour, 21 minutes
                  1h 21m