Uploaded image for project: 'Percona Server for MySQL'
  1. Percona Server for MySQL
  2. PS-3826

Failing assertion: index->table->stat_initialized

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Priority: High
    • Resolution: Cannot Reproduce
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Running Percona Server 5.6.38-rel83.0 on a CentOS 6 x86_64 server with 96 GB RAM and about 200K InnoDB tables (that's not a typo) plus a handful of MyISAM ones.

      Every week or so, we get a situation where the MySQL server crashes, leaving this backtrace:

       

       

      2018-02-12 21:19:27 7fbdb74a2700 InnoDB: Assertion failure in thread 140452800636672 in file ha_innodb.cc line 12153
      InnoDB: Failing assertion: index->table->stat_initialized
      InnoDB: We intentionally generate a memory trap.
      InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
      InnoDB: If you get repeated assertion failures or crashes, even
      InnoDB: immediately after the mysqld startup, there may be
      InnoDB: corruption in the InnoDB tablespace. Please refer to
      InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
      InnoDB: about forcing recovery.
      02:19:28 UTC - mysqld got signal 6 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
      We will try our best to scrape up some info that will hopefully help
      diagnose the problem, but since we have already crashed,
      something is definitely wrong and this may fail.
      Please help us make Percona Server better by reporting any
      bugs at http://bugs.percona.com/
      
      key_buffer_size=33554432
      read_buffer_size=2097152
      max_used_connections=1210
      max_threads=1502
      thread_count=164
      connection_count=164
      It is possible that mysqld could use up to
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6205868 K bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
      
      Thread pointer: 0x7fbd7c8c7000
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 7fbdb74a1d30 thread_stack 0x40000
      /usr/sbin/mysqld(my_print_stacktrace+0x2c)[0x8db9bc]
      /usr/sbin/mysqld(handle_fatal_signal+0x461)[0x65b821]
      /lib64/libpthread.so.0(+0xf7e0)[0x7fd1572917e0]
      /lib64/libc.so.6(gsignal+0x35)[0x7fd1554ca495]
      /lib64/libc.so.6(abort+0x175)[0x7fd1554cbc75]
      /usr/sbin/mysqld[0x994015]
      /usr/sbin/mysqld[0x9984f5]
      /usr/sbin/mysqld(_ZN7handler7ha_openEP5TABLEPKcii+0x33)[0x59e663]
      /usr/sbin/mysqld(_Z21open_table_from_shareP3THDP11TABLE_SHAREPKcjjjP5TABLEb+0x694)[0x763d74]
      /usr/sbin/mysqld(_Z10open_tableP3THDP10TABLE_LISTP18Open_table_context+0x1116)[0x690d56]
      /usr/sbin/mysqld(_Z11open_tablesP3THDPP10TABLE_LISTPjjP19Prelocking_strategy+0x6c5)[0x698b95]
      /usr/sbin/mysqld(_Z30open_normal_and_derived_tablesP3THDP10TABLE_LISTj+0x51)[0x699471]
      /usr/sbin/mysqld[0x55e324]
      /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1b9e)[0x6df10e]
      /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x5b8)[0x6e4848]
      /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x100e)[0x6e5fee]
      /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x1a2)[0x6b2682]
      /usr/sbin/mysqld(handle_one_connection+0x40)[0x6b2720]
      /lib64/libpthread.so.0(+0x7aa1)[0x7fd157289aa1]
      /lib64/libc.so.6(clone+0x6d)[0x7fd155580bcd]
      
      Trying to get some variables.
      Some pointers may be invalid and cause the dump to abort.
      Query (7fbd8c880010): is an invalid pointer
      Connection ID (thread ID): 32018292
      Status: NOT_KILLED
      
      
      

       

      When this happens fortunately the server restarts, but it creates a ≈20 minute downtime which is not that neat.  Attached is the my.cnf running on that server.

       

      Not sure what to do to make this easier to debug for you guys.  I could not find the issue on Oracle's bugtracker; let me know if I should bring the issue upstream.

       

      Thanks!

       

        Attachments

        1. my.cnf
          3 kB
        2. PS-3826.patch
          0.5 kB

          Activity

            People

            Assignee:
            sveta.smirnova Sveta Smirnova
            Reporter:
            p.levesque@nexalogy.com Patrice Levesque
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Smart Checklist