Uploaded image for project: 'Percona Server for MySQL'
  1. Percona Server for MySQL
  2. PS-7010

Modify to Lock buffer blocks before sanity check in btr_cur_latch_leaves

Details

    • Bug
    • Status: Done
    • Medium
    • Resolution: Fixed
    • 5.7.28-31
    • 5.7.31-34, 8.0.21-12
    • None
    • Yes

    Description

      Crash seems consistent with this upstream bug https://bugs.mysql.com/bug.php?id=74596

      We had 2 crashes with instances running Percona Server 5.7.28-31, this is the stack trace:

      2020-04-26 00:19:19 0x7f7985979700  InnoDB: Assertion failure in thread 140159909074688 in file btr0cur.cc line 402
      InnoDB: Failing assertion: page_is_comp(get_block->frame) == page_is_comp(page)
      InnoDB: We intentionally generate a memory trap.
      InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
      InnoDB: If you get repeated assertion failures or crashes, even
      InnoDB: immediately after the mysqld startup, there may be
      InnoDB: corruption in the InnoDB tablespace. Please refer to
      InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
      InnoDB: about forcing recovery.
      00:19:19 UTC - mysqld got signal 6 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
      Attempting to collect some information that could help diagnose the problem.
      As this is a crash and something is definitely wrong, the information
      collection process might fail.
      Please help us make Percona Server better by reporting any
      bugs at https://bugs.percona.com/
      
      key_buffer_size=33554432
      read_buffer_size=2097152
      max_used_connections=2241
      max_threads=8021
      thread_count=117
      connection_count=117
      It is possible that mysqld could use up to
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 82271616 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
      
      Thread pointer: 0x7f5e13016000
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 7f7985978e30 thread_stack 0x40000
      __mysqld(my_print_stacktrace+0x3b)[0xe9e33b]
      __mysqld(handle_fatal_signal+0x489)[0xcc7d49]
      libpthread.so.0(+0x11390)[0x7fa3f49ec390]
      libc.so.6(gsignal+0x38)[0x7fa3f2882428]
      libc.so.6(abort+0x16a)[0x7fa3f288402a]
      __mysqld[0x7557d8]
      __mysqld(_Z20btr_cur_latch_leavesP11buf_block_tRK9page_id_tRK11page_size_tmP9btr_cur_tP5mtr_t+0x718)[0x1169658]
      __mysqld(_Z27btr_cur_search_to_nth_levelP12dict_index_tmPK8dtuple_t15page_cur_mode_tmP9btr_cur_tmPKcmP5mtr_t+0x1c61)[0x1171ef1]
      __mysqld(_Z30btr_pcur_restore_position_funcmP10btr_pcur_tPKcmP5mtr_t+0x409)[0x1178399]
      __mysqld(_Z32btr_pcur_move_backward_from_pageP10btr_pcur_tP5mtr_t+0x81)[0x11788c1]
      __mysqld(_Z21btr_pcur_move_to_prevP10btr_pcur_tP5mtr_t+0x4a)[0x1178b4a]
      __mysqld(_Z15row_search_mvccPh15page_cur_mode_tP14row_prebuilt_tmm+0x124f)[0x10bb28f]
      __mysqld(_ZN11ha_innobase10index_readEPhPKhj16ha_rkey_function+0x30a)[0xf99afa]
      __mysqld(_ZN7handler17ha_index_read_mapEPhPKhm16ha_rkey_function+0x3c1)[0x78a7a1]
      __mysqld(_ZN17QUICK_SELECT_DESC8get_nextEv+0x18b)[0xd9be4b]
      __mysqld[0xb439cf]
      __mysqld(_Z10sub_selectP4JOINP7QEP_TABb+0x112)[0xbb97c2]
      __mysqld(_ZN4JOIN4execEv+0x3c8)[0xbb2388]
      __mysqld(_ZN18st_select_lex_unit7executeEP3THD+0x1ec)[0xc6e69c]
      __mysqld(_Z12handle_queryP3THDP3LEXP12Query_resultyy+0x2d5)[0xc26c85]
      __mysqld[0x747d89]
      __mysqld(_Z21mysql_execute_commandP3THDb+0x4593)[0xbe5f23]
      __mysqld(_Z11mysql_parseP3THDP12Parser_stateb+0x4b5)[0xbe9555]
      __mysqld(_Z16dispatch_commandP3THDPK8COM_DATA19enum_server_command+0x119f)[0xbea7ff]
      __mysqld(_Z10do_commandP3THD+0x1df)[0xbebdef]
      __mysqld(handle_connection+0x2c0)[0xcae140]
      __mysqld(pfs_spawn_thread+0x1b4)[0xeb8994]
      libpthread.so.0(+0x76ba)[0x7fa3f49e26ba]
      libc.so.6(clone+0x6d)[0x7fa3f295441d]

      Related table:

      CREATE TABLE `45_21_kv` (
        `key` varbinary(742) NOT NULL,
        `version` bigint(20) NOT NULL,
        `value` mediumblob,
        `pelock` binary(24) DEFAULT NULL,
        PRIMARY KEY (`key`,`version`)
      ) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci ROW_FORMAT=COMPRESSED KEY_BLOCK_SIZE=4
      1 row in set (0.00 sec)

      Related variables:

      innodb_buffer_pool_filename = ib_buffer_pool
      innodb_buffer_pool_instances = 8
      innodb_buffer_pool_size = 260G
      innodb_change_buffering = all
      innodb_data_file_path = ibdata1:10M:autoextend
      innodb_fast_shutdown = 0
      innodb_file_format = barracuda
      innodb_file_per_table
      innodb_flush_log_at_trx_commit = 1
      innodb_flush_method = O_DIRECT
      innodb_flush_neighbors = 0
      innodb_io_capacity = 60000
      innodb_lock_wait_timeout = 5
      innodb_log_buffer_size = 8M
      innodb_log_file_size = 1G
      innodb_log_files_in_group = 3
      innodb_lru_scan_depth = 5000
      innodb_page_size = 16384
      

      We can't see issues with instances running same workload but not using compressed tables. Also not reproducible in 5.6

      Attachments

        Activity

          People

            Unassigned Unassigned
            francisco.bordenave Francisco Bordenave
            Votes:
            1 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - 2 hours, 30 minutes Original Estimate - 2 hours, 30 minutes
                2h 30m
                Remaining:
                Time Spent - 1 day, 4 hours, 19 minutes Remaining Estimate - 2 hours, 42 minutes
                2h 42m
                Logged:
                Time Spent - 1 day, 4 hours, 19 minutes Remaining Estimate - 2 hours, 42 minutes
                1d 4h 19m

                Smart Checklist