Description
Crash seems consistent with this upstream bug https://bugs.mysql.com/bug.php?id=74596
We had 2 crashes with instances running Percona Server 5.7.28-31, this is the stack trace:
2020-04-26 00:19:19 0x7f7985979700 InnoDB: Assertion failure in thread 140159909074688 in file btr0cur.cc line 402 InnoDB: Failing assertion: page_is_comp(get_block->frame) == page_is_comp(page) InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 00:19:19 UTC - mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. Attempting to collect some information that could help diagnose the problem. As this is a crash and something is definitely wrong, the information collection process might fail. Please help us make Percona Server better by reporting any bugs at https://bugs.percona.com/ key_buffer_size=33554432 read_buffer_size=2097152 max_used_connections=2241 max_threads=8021 thread_count=117 connection_count=117 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 82271616 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f5e13016000 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 7f7985978e30 thread_stack 0x40000 __mysqld(my_print_stacktrace+0x3b)[0xe9e33b] __mysqld(handle_fatal_signal+0x489)[0xcc7d49] libpthread.so.0(+0x11390)[0x7fa3f49ec390] libc.so.6(gsignal+0x38)[0x7fa3f2882428] libc.so.6(abort+0x16a)[0x7fa3f288402a] __mysqld[0x7557d8] __mysqld(_Z20btr_cur_latch_leavesP11buf_block_tRK9page_id_tRK11page_size_tmP9btr_cur_tP5mtr_t+0x718)[0x1169658] __mysqld(_Z27btr_cur_search_to_nth_levelP12dict_index_tmPK8dtuple_t15page_cur_mode_tmP9btr_cur_tmPKcmP5mtr_t+0x1c61)[0x1171ef1] __mysqld(_Z30btr_pcur_restore_position_funcmP10btr_pcur_tPKcmP5mtr_t+0x409)[0x1178399] __mysqld(_Z32btr_pcur_move_backward_from_pageP10btr_pcur_tP5mtr_t+0x81)[0x11788c1] __mysqld(_Z21btr_pcur_move_to_prevP10btr_pcur_tP5mtr_t+0x4a)[0x1178b4a] __mysqld(_Z15row_search_mvccPh15page_cur_mode_tP14row_prebuilt_tmm+0x124f)[0x10bb28f] __mysqld(_ZN11ha_innobase10index_readEPhPKhj16ha_rkey_function+0x30a)[0xf99afa] __mysqld(_ZN7handler17ha_index_read_mapEPhPKhm16ha_rkey_function+0x3c1)[0x78a7a1] __mysqld(_ZN17QUICK_SELECT_DESC8get_nextEv+0x18b)[0xd9be4b] __mysqld[0xb439cf] __mysqld(_Z10sub_selectP4JOINP7QEP_TABb+0x112)[0xbb97c2] __mysqld(_ZN4JOIN4execEv+0x3c8)[0xbb2388] __mysqld(_ZN18st_select_lex_unit7executeEP3THD+0x1ec)[0xc6e69c] __mysqld(_Z12handle_queryP3THDP3LEXP12Query_resultyy+0x2d5)[0xc26c85] __mysqld[0x747d89] __mysqld(_Z21mysql_execute_commandP3THDb+0x4593)[0xbe5f23] __mysqld(_Z11mysql_parseP3THDP12Parser_stateb+0x4b5)[0xbe9555] __mysqld(_Z16dispatch_commandP3THDPK8COM_DATA19enum_server_command+0x119f)[0xbea7ff] __mysqld(_Z10do_commandP3THD+0x1df)[0xbebdef] __mysqld(handle_connection+0x2c0)[0xcae140] __mysqld(pfs_spawn_thread+0x1b4)[0xeb8994] libpthread.so.0(+0x76ba)[0x7fa3f49e26ba] libc.so.6(clone+0x6d)[0x7fa3f295441d]
Related table:
CREATE TABLE `45_21_kv` ( `key` varbinary(742) NOT NULL, `version` bigint(20) NOT NULL, `value` mediumblob, `pelock` binary(24) DEFAULT NULL, PRIMARY KEY (`key`,`version`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci ROW_FORMAT=COMPRESSED KEY_BLOCK_SIZE=4 1 row in set (0.00 sec)
Related variables:
innodb_buffer_pool_filename = ib_buffer_pool innodb_buffer_pool_instances = 8 innodb_buffer_pool_size = 260G innodb_change_buffering = all innodb_data_file_path = ibdata1:10M:autoextend innodb_fast_shutdown = 0 innodb_file_format = barracuda innodb_file_per_table innodb_flush_log_at_trx_commit = 1 innodb_flush_method = O_DIRECT innodb_flush_neighbors = 0 innodb_io_capacity = 60000 innodb_lock_wait_timeout = 5 innodb_log_buffer_size = 8M innodb_log_file_size = 1G innodb_log_files_in_group = 3 innodb_lru_scan_depth = 5000 innodb_page_size = 16384
We can't see issues with instances running same workload but not using compressed tables. Also not reproducible in 5.6