Uploaded image for project: 'Percona Server for MySQL'
  1. Percona Server for MySQL
  2. PS-311

On server shutdown threadpool is crashing in debug mode due to race condition

Details

    • Bug
    • Status: Done
    • High
    • Resolution: Fixed
    • 5.7.x
    • None
    • None

    Description

      There is a race condition that causes debug server to crash. The race is between main server thread and main worker thread. The stack-trace:

      15:04:24 Program terminated with signal SIGABRT, Aborted.
       15:04:24 #0  0x00007f9341966c37 in __GI_raise ([email protected]=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
       15:04:24 56    ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
       15:04:24 #0  0x00007f9341966c37 in __GI_raise ([email protected]=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
       15:04:24 #1  0x00007f934196a028 in __GI_abort () at abort.c:89
       15:04:24 #2  0x00007f934195fbf6 in _assert_fail_base (fmt=0x7f9341ab4018 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", [email protected]=0x1e24536 "THR_THD_initialized", [email protected]=0x1e244e0 "/mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/sql/mysqld.h", [email protected]=392, [email protected]=0x1e24fa0 <_ZZL21my_thread_get_THR_THDvE19PRETTY_FUNCTION_> "THD* my_thread_get_THR_THD()") at assert.c:92
       15:04:24 #3  0x00007f934195fca2 in _GI_assert_fail (assertion=0x1e24536 "THR_THD_initialized", file=0x1e244e0 "/mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/sql/mysqld.h", line=392, function=0x1e24fa0 <_ZZL21my_thread_get_THR_THDvE19PRETTY_FUNCTION_> "THD* my_thread_get_THR_THD()") at assert.c:101
       15:04:24 #4  0x0000000000e8edfc in my_thread_get_THR_THD () at /mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/sql/mysqld.h:392
       15:04:24 #5  0x0000000000e8f1b6 in handle_fatal_signal (sig=6) at /mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/sql/signal_handler.cc:140
       15:04:24 #6  <signal handler called>
       15:04:24 #7  0x00007f9341966c37 in __GI_raise ([email protected]=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
       15:04:24 #8  0x00007f934196a028 in __GI_abort () at abort.c:89
       15:04:24 #9  0x00007f934195fbf6 in _assert_fail_base (fmt=0x7f9341ab4018 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", [email protected]=0x2214b8d "THR_KEY_mysys_initialized", [email protected]=0x2214b30 "/mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/mysys/my_thr_init.c", [email protected]=75, [email protected]=0x2214c80 <PRETTY_FUNCTION_.11273> "set_mysys_thread_var") at assert.c:92
       15:04:24 #10 0x00007f934195fca2 in _GI_assert_fail (assertion=0x2214b8d "THR_KEY_mysys_initialized", file=0x2214b30 "/mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/mysys/my_thr_init.c", line=75, function=0x2214c80 <PRETTY_FUNCTION_.11273> "set_mysys_thread_var") at assert.c:101
       15:04:24 #11 0x000000000189c25d in set_mysys_thread_var (mysys_var=0x0) at /mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/mysys/my_thr_init.c:75
       15:04:24 #12 0x000000000189cb90 in my_thread_end () at /mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/mysys/my_thr_init.c:370
       15:04:24 #13 0x0000000001673f07 in worker_main (param=0x2d61a00 <all_groups>) at /mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/sql/threadpool_unix.cc:1651
       15:04:24 #14 0x0000000001ce9361 in pfs_spawn_thread (arg=0x42e34a0) at /mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/storage/perfschema/pfs.cc:2190
       15:04:24 #15 0x00007f9342531184 in start_thread (arg=0x7f9333fbe700) at pthread_create.c:312
       15:04:24 #16 0x00007f9341a2dffd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
       15:04:24 
       15:04:24 Thread 2 (Thread 0x7f9343dfd780 (LWP 24033)):
       15:04:24 #0  os_event::~os_event (this=0x190fed0 <GenericPolicy<TTASEventMutex<GenericPolicy> >::~GenericPolicy()+42>, __in_chrg=<optimized out>) at /mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/storage/innobase/os/os0event.cc:238
       15:04:24 #1  0x000000000190d338 in TTASEventMutex<GenericPolicy>::~TTASEventMutex (this=0x2d7bd20 <ibuf_bitmap_mutex>, __in_chrg=<optimized out>) at /mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/storage/innobase/include/ib0mutex.h:627
       15:04:24 #2  0x000000000190f5a2 in PolicyMutex<TTASEventMutex<GenericPolicy> >::~PolicyMutex (this=0x2d7bd20 <ibuf_bitmap_mutex>, __in_chrg=<optimized out>) at /mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/storage/innobase/include/ib0mutex.h:926
       15:04:24 #3  0x00007f934196c1a9 in _run_exit_handlers (status=0, listp=0x7f9341cf26c8 <_exit_funcs>, run_list_atexit[email protected]=true) at exit.c:82
       15:04:24 #4  0x00007f934196c1f5 in __GI_exit (status=<optimized out>) at exit.c:104
       15:04:24 #5  0x0000000000e77e99 in mysqld_exit (exit_code=0) at /mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/sql/mysqld.cc:1206
       15:04:24 #6  0x0000000000e80060 in mysqld_main (argc=94, argv=0x392bd58) at /mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/sql/mysqld.cc:5445
       15:04:24 #7  0x0000000000e7637d in main (argc=9, argv=0x7fffe391c5b8) at /mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/sql/main.cc:25
       15:04:24 
       15:04:24 Thread 1 (Thread 0x7f9333fbe700 (LWP 24116)):
       15:04:24 #0  0x00007f9341966c37 in __GI_raise ([email protected]=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
       15:04:24 #1  0x00007f934196a028 in __GI_abort () at abort.c:89
       15:04:24 #2  0x00007f934195fbf6 in _assert_fail_base (fmt=0x7f9341ab4018 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", [email protected]=0x1e24536 "THR_THD_initialized", [email protected]=0x1e244e0 "/mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/sql/mysqld.h", [email protected]=392, [email protected]=0x1e24fa0 <_ZZL21my_thread_get_THR_THDvE19PRETTY_FUNCTION_> "THD* my_thread_get_THR_THD()") at assert.c:92
       15:04:24 #3  0x00007f934195fca2 in _GI_assert_fail (assertion=0x1e24536 "THR_THD_initialized", file=0x1e244e0 "/mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/sql/mysqld.h", line=392, function=0x1e24fa0 <_ZZL21my_thread_get_THR_THDvE19PRETTY_FUNCTION_> "THD* my_thread_get_THR_THD()") at assert.c:101
       15:04:24 #4  0x0000000000e8edfc in my_thread_get_THR_THD () at /mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/sql/mysqld.h:392
       15:04:24 #5  0x0000000000e8f1b6 in handle_fatal_signal (sig=6) at /mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/sql/signal_handler.cc:140
       15:04:24 #6  <signal handler called>
       15:04:24 #7  0x00007f9341966c37 in __GI_raise ([email protected]=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
       15:04:24 #8  0x00007f934196a028 in __GI_abort () at abort.c:89
       15:04:24 #9  0x00007f934195fbf6 in _assert_fail_base (fmt=0x7f9341ab4018 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", [email protected]=0x2214b8d "THR_KEY_mysys_initialized", [email protected]=0x2214b30 "/mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/mysys/my_thr_init.c", [email protected]=75, [email protected]=0x2214c80 <PRETTY_FUNCTION_.11273> "set_mysys_thread_var") at assert.c:92
       15:04:24 #10 0x00007f934195fca2 in _GI_assert_fail (assertion=0x2214b8d "THR_KEY_mysys_initialized", file=0x2214b30 "/mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/mysys/my_thr_init.c", line=75, function=0x2214c80 <PRETTY_FUNCTION_.11273> "set_mysys_thread_var") at assert.c:101
       15:04:24 #11 0x000000000189c25d in set_mysys_thread_var (mysys_var=0x0) at /mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/mysys/my_thr_init.c:75
       15:04:24 #12 0x000000000189cb90 in my_thread_end () at /mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/mysys/my_thr_init.c:370
       15:04:24 #13 0x0000000001673f07 in worker_main (param=0x2d61a00 <all_groups>) at /mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/sql/threadpool_unix.cc:1651
       15:04:24 #14 0x0000000001ce9361 in pfs_spawn_thread (arg=0x42e34a0) at /mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/ubuntu-trusty-64bit/storage/perfschema/pfs.cc:2190
       15:04:24 #15 0x00007f9342531184 in start_thread (arg=0x7f9333fbe700) at pthread_create.c:312
       15:04:24 #16 0x00007f9341a2dffd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

      Attachments

        Activity

          People

            robert.golebiowski Robert Golebiowski (Inactive)
            robert.golebiowski Robert Golebiowski (Inactive)
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0 minutes
                0m
                Logged:
                Time Spent - 1 hour, 21 minutes
                1h 21m

                Smart Checklist