Details
-
Bug
-
Status: Done
-
Medium
-
Resolution: Incomplete
-
5.7.24-26
-
None
-
None
Description
Seen the following crash on multiple servers that were upgraded from 5.6 and are now running 5.7.24 with innodb_track_changed_pages=1.
2019-06-05T12:50:02.122869Z 0 [ERROR] InnoDB: Log block checksum mismatch: LSN 33890770911232, expected 2870857039, calculated checksum 969592942, stored log block n:o 694660673, expected log block n:o 694660673 2019-06-05 05:50:02 0x7f63e77f7700 InnoDB: Assertion failure in thread 140067062380288 in file log0online.cc line 946 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 12:50:02 UTC - mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. Attempting to collect some information that could help diagnose the problem. As this is a crash and something is definitely wrong, the information collection process might fail. Please help us make Percona Server better by reporting any bugs at http://bugs.percona.com/ key_buffer_size=8388608 read_buffer_size=131072 max_used_connections=143 max_threads=1001 thread_count=7 connection_count=7 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1174370 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0 thread_stack 0x40000 /usr/sbin/mysqld(my_print_stacktrace+0x2c)[0xedf27c] /usr/sbin/mysqld(handle_fatal_signal+0x461)[0xd206f1] /lib64/libpthread.so.0[0x3f6e80f7e0] /lib64/libc.so.6(gsignal+0x35)[0x3f6e032495] /lib64/libc.so.6(abort+0x175)[0x3f6e033c75] /usr/sbin/mysqld[0x777028] /usr/sbin/mysqld[0x103defa] /usr/sbin/mysqld(_Z26log_online_follow_redo_logv+0x400)[0x1041090] /usr/sbin/mysqld(srv_redo_log_follow_thread+0xbd)[0x111de0d] /lib64/libpthread.so.0[0x3f6e807aa1] /lib64/libc.so.6(clone+0x6d)[0x3f6e0e8bcd] You may download the Percona Server operations manual by visiting http://www.percona.com/software/percona-server/. You may find information in the manual which will help you identify the cause of the crash.
Checksum format being used is crc32:
Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 8627 Server version: 5.7.24-27-log Percona Server (GPL), Release 27, Revision bd42700 Copyright (c) 2009-2018 Percona LLC and/or its affiliates Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> show variables like '%checksum%'; +---------------------------+-------+ | Variable_name | Value | +---------------------------+-------+ | binlog_checksum | CRC32 | | innodb_checksum_algorithm | crc32 | | innodb_checksums | ON | | innodb_log_checksums | ON | | master_verify_checksum | OFF | | slave_sql_verify_checksum | ON | +---------------------------+-------+ 6 rows in set (0.01 sec)
Further debug info below:
# resolve_stack_dump -s /tmp/mysqld.sym -n a 0xedf27c my_print_stacktrace + 44 0xd206f1 handle_fatal_signal + 1121 0x3f6e80f7e0 _end + 1822088024 0x3f6e032495 _end + 1813841933 0x3f6e033c75 _end + 1813848045 0x777028 _ZStlsISt11char_traitsIcEERSt13basic_ostreamIcT_ES5_PKc.part.1 + 0 0x103defa _ZL27log_online_is_valid_log_segPKhm + 522 0x1041090 _Z26log_online_follow_redo_logv + 1024 0x111de0d srv_redo_log_follow_thread + 189 0x3f6e807aa1 _end + 1822055961 0x3f6e0e8bcd _end + 1814589253 # addr2line -fie /usr/lib/debug/usr/sbin/mysqld.debug 0x777028 operator<< <std::char_traits<char> > /opt/percona-devtoolset/root/usr/include/c++/4.8.2/ostream:530 # addr2line -fie /usr/lib/debug/usr/sbin/mysqld.debug 0x103defa log_online_is_valid_log_seg /usr/src/debug/percona-server-5.7.24-27/percona-server-5.7.24-27/storage/innobase/log/log0online.cc:946 # addr2line -fie /usr/lib/debug/usr/sbin/mysqld.debug 0x1041090 log_online_follow_log_seg /usr/src/debug/percona-server-5.7.24-27/percona-server-5.7.24-27/storage/innobase/log/log0online.cc:1041 log_online_follow_log_group /usr/src/debug/percona-server-5.7.24-27/percona-server-5.7.24-27/storage/innobase/log/log0online.cc:1099 _Z26log_online_follow_redo_logv /usr/src/debug/percona-server-5.7.24-27/percona-server-5.7.24-27/storage/innobase/log/log0online.cc:1304 # addr2line -fie /usr/lib/debug/usr/sbin/mysqld.debug 0x111de0d srv_redo_log_follow_thread /usr/src/debug/percona-server-5.7.24-27/percona-server-5.7.24-27/storage/innobase/srv/srv0srv.cc:2245
Digging through the code, seems the log_block_get_checksum() and log_block_calc_checksum() functions are returning non-matching values in log0recv.cc#L661 hence causing the assert