Uploaded image for project: 'Percona Server for MySQL'
  1. Percona Server for MySQL
  2. PS-4711

crash on TokuDB PFS-instrumented mutexes deinitialization

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Priority: Medium
    • Resolution: Fixed
    • Affects Version/s: 5.7.22-22
    • Fix Version/s: 5.7.23-23, 8.0.13-3
    • Component/s: TokuDB
    • Labels:
      None

      Description

      1) tokudb_map_mutex is global object which is initialized in tokudb_init_func(). The object contains the pointer to data inside of PFS plugin(tokudb_map_mutex._mutex.m_psi). This data is initialized in pfs_init_mutex_v1(). To avoid memory allocation/deallocation on PFS calls global containers are used, in particular see global_mutex_container, mutex_class_array and pfs_buffer_container.*. The backtrace of tokudb_map_mutex._mutex.m_psi initialization is the following:

      (gdb) bt
      #0  PFS_buffer_scalable_container<PFS_mutex, 1024, 1024, PFS_buffer_default_array<PFS_mutex>, PFS_buffer_default_allocator<PFS_mutex> >::allocate (
          this=0x2f991b0, dirty_state=0x7fffffff5c20)
          at ./storage/perfschema/pfs_buffer_container.h:565
      #1  0x0000000001989bee in PFS_partitioned_buffer_scalable_container<PFS_buffer_scalable_container<PFS_mutex, 1024, 1024, PFS_buffer_default_array<PFS_mutex>, PFS_buffer_default_allocator<PFS_mutex> >, 2>::allocate (this=0x2f06030,
          dirty_state=0x7fffffff5c20, partition=0)
          at ./storage/perfschema/pfs_buffer_container.h:1177
      #2  0x0000000001983685 in create_mutex (klass=0x2fb32c0,
          identity=0x7fffe3a8e3e0)
          at ./storage/perfschema/pfs_instr.cc:298
      #3  0x00000000019662e3 in pfs_init_mutex_v1 (key=225, identity=0x7fffe3a8e3e0)
          at ./storage/perfschema/pfs.cc:1820
      #4  0x00007fffe3693cf6 in inline_mysql_mutex_init (key=225,
          that=0x7fffe3a8e3e0, attr=0x2f90670,
          src_file=0x7fffe37e0330 "./storage/tokudb/tokudb_thread.h", src_line=224)
          at ./include/mysql/psi/mysql_thread.h:660
      #5  0x00007fffe36cfd91 in tokudb::thread::mutex_t::reinit (
          this=0x7fffe3a8e3e0, key=225)
          at ./storage/tokudb/tokudb_thread.h:224
      #6  0x00007fffe36c7229 in tokudb_init_func (p=0x3598d20)
          at ./storage/tokudb/hatoku_hton.cc:283
      #7  0x0000000000fbfc6d in ha_initialize_handlerton (plugin=0x357f7b0)
          at ./sql/handler.cc:871
      #8  0x0000000001623c66 in plugin_initialize (plugin=0x357f7b0)
          at ./sql/sql_plugin.cc:1252
      #9  0x000000000162405c in plugin_init_initialize_and_reap ()
          at ./sql/sql_plugin.cc:1407
      #10 0x0000000001624c5b in plugin_register_dynamic_and_init_all (
          argc=0x2eb8530, argv=0x2fa14a8, flags=0)
          at ./sql/sql_plugin.cc:1692
      #11 0x0000000000f1ca5e in init_server_components ()
          at ./sql/mysqld.cc:4360
      #12 0x0000000000f1e2b3 in mysqld_main (argc=102, argv=0x2fa14a8)
          at ./sql/mysqld.cc:5124
      #13 0x0000000000f14fec in main (argc=17, argv=0x7fffffff6c48)
          at ./sql/main.cc:25
      

      2) If PFS is deinitialized before tokudb storage engine

      #0  PFS_buffer_scalable_container<PFS_mutex, 1024, 1024, PFS_buffer_default_array<PFS_mutex>, PFS_buffer_default_allocator<PFS_mutex> >::cleanup (
          this=0x2f991b0)
          at ./storage/perfschema/pfs_buffer_container.h:488
      #1  0x00000000019895a3 in PFS_partitioned_buffer_scalable_container<PFS_buffer_scalable_container<PFS_mutex, 1024, 1024, PFS_buffer_default_array<PFS_mutex>, PFS_buffer_default_allocator<PFS_mutex> >, 2>::cleanup (this=0x2f06030)
          at ./storage/perfschema/pfs_buffer_container.h:1128
      #2  0x000000000198339b in cleanup_instruments ()
          at ./storage/perfschema/pfs_instr.cc:209
      #3  0x0000000001994286 in cleanup_performance_schema ()
          at ./storage/perfschema/pfs_server.cc:289
      #4  0x0000000001994368 in shutdown_performance_schema ()
          at ./storage/perfschema/pfs_server.cc:319
      #5  0x0000000000f16c56 in mysqld_exit (exit_code=0)
          at ./sql/mysqld.cc:1217
      #6  0x0000000000f1ef26 in mysqld_main (argc=102, argv=0x2fa14a8)
      ---Type <return> to continue, or q <return> to quit---
          at ./sql/mysqld.cc:5559
      #7  0x0000000000f14fec in main (argc=17, argv=0x7fffffff6c48)
          at ./sql/main.cc:25
      

       we will have segmentation fault on tokudb_map_mutex(and, more likely, on other global mutex objects) deinitialization with the following backtrace:

      Program received signal SIGSEGV, Segmentation fault.
      0x0000000001983788 in destroy_mutex (pfs=0x7ffff11120c0)
          at ./storage/perfschema/pfs_instr.cc:323
      323       PFS_mutex_class *klass= pfs->m_class;
      (gdb) bt
      #0  0x0000000001983788 in destroy_mutex (pfs=0x7ffff11120c0)
          at ./storage/perfschema/pfs_instr.cc:323
      #1  0x0000000001966333 in pfs_destroy_mutex_v1 (mutex=0x7ffff11120c0)
          at ./storage/perfschema/pfs.cc:1833
      #2  0x00007fffe3693d55 in inline_mysql_mutex_destroy (that=0x7fffe3a8e3e0,
          src_file=0x7fffe37e0330 "./storage/tokudb/tokudb_thread.h", src_line=214)
          at ./include/mysql/psi/mysql_thread.h:681
      #3  0x00007fffe36cfcae in tokudb::thread::mutex_t::~mutex_t (
          this=0x7fffe3a8e3e0, __in_chrg=<optimized out>)
          at ./storage/tokudb/tokudb_thread.h:214
      #4  0x00007ffff5e09aa2 in __run_exit_handlers (status=0, listp=0x7ffff615a4c8,
          run_list_atexit=true) at exit.c:78
      #5  0x00007ffff5e09af5 in *__GI_exit (status=-250535744) at exit.c:100
      #6  0x0000000000f16c60 in mysqld_exit (exit_code=0)
          at ./sql/mysqld.cc:1231
      #7  0x0000000000f1ef26 in mysqld_main (argc=102, argv=0x2fa14a8)
          at ./sql/mysqld.cc:5559
      #8  0x0000000000f14fec in main (argc=17, argv=0x7fffffff6c48)
          at ./sql/main.cc:25
      

        Smart Checklist

          Attachments

            Activity

              People

              Assignee:
              vladislav.lesin Vladislav Lesin (Inactive)
              Reporter:
              vladislav.lesin Vladislav Lesin (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 1 day, 6 hours, 20 minutes
                  1d 6h 20m