Uploaded image for project: 'Percona Server for MySQL'
  1. Percona Server for MySQL
  2. PS-5676

Ocassional crashes due to Log block checksum mismatch when innodb_track_changed_pages=1

Details

    • Bug
    • Status: Done
    • Medium
    • Resolution: Incomplete
    • 5.7.24-26
    • None
    • None

    Description

      Seen the following crash on multiple servers that were upgraded from 5.6 and are now running 5.7.24 with innodb_track_changed_pages=1.

      2019-06-05T12:50:02.122869Z 0 [ERROR] InnoDB: Log block checksum mismatch: LSN 33890770911232, expected 2870857039, calculated checksum 969592942, stored log block n:o 694660673, expected log block n:o 694660673
      2019-06-05 05:50:02 0x7f63e77f7700  InnoDB: Assertion failure in thread 140067062380288 in file log0online.cc line 946
      InnoDB: We intentionally generate a memory trap.
      InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
      InnoDB: If you get repeated assertion failures or crashes, even
      InnoDB: immediately after the mysqld startup, there may be
      InnoDB: corruption in the InnoDB tablespace. Please refer to
      InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
      InnoDB: about forcing recovery.
      12:50:02 UTC - mysqld got signal 6 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
      Attempting to collect some information that could help diagnose the problem.
      As this is a crash and something is definitely wrong, the information
      collection process might fail.
      Please help us make Percona Server better by reporting any
      bugs at http://bugs.percona.com/
      
      key_buffer_size=8388608
      read_buffer_size=131072
      max_used_connections=143
      max_threads=1001
      thread_count=7
      connection_count=7
      It is possible that mysqld could use up to
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1174370 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
      
      Thread pointer: 0x0
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0 thread_stack 0x40000
      /usr/sbin/mysqld(my_print_stacktrace+0x2c)[0xedf27c]
      /usr/sbin/mysqld(handle_fatal_signal+0x461)[0xd206f1]
      /lib64/libpthread.so.0[0x3f6e80f7e0]
      /lib64/libc.so.6(gsignal+0x35)[0x3f6e032495]
      /lib64/libc.so.6(abort+0x175)[0x3f6e033c75]
      /usr/sbin/mysqld[0x777028]
      /usr/sbin/mysqld[0x103defa]
      /usr/sbin/mysqld(_Z26log_online_follow_redo_logv+0x400)[0x1041090]
      /usr/sbin/mysqld(srv_redo_log_follow_thread+0xbd)[0x111de0d]
      /lib64/libpthread.so.0[0x3f6e807aa1]
      /lib64/libc.so.6(clone+0x6d)[0x3f6e0e8bcd]
      You may download the Percona Server operations manual by visiting
      http://www.percona.com/software/percona-server/. You may find information
      in the manual which will help you identify the cause of the crash.
      

      Checksum format being used is crc32:

      Welcome to the MySQL monitor.  Commands end with ; or \g.
      Your MySQL connection id is 8627
      Server version: 5.7.24-27-log Percona Server (GPL), Release 27, Revision bd42700
      
      Copyright (c) 2009-2018 Percona LLC and/or its affiliates
      Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.
      
      Oracle is a registered trademark of Oracle Corporation and/or its
      affiliates. Other names may be trademarks of their respective
      owners.
      
      Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
      
      mysql> show variables like '%checksum%';
      +---------------------------+-------+
      | Variable_name             | Value |
      +---------------------------+-------+
      | binlog_checksum           | CRC32 |
      | innodb_checksum_algorithm | crc32 |
      | innodb_checksums          | ON    |
      | innodb_log_checksums      | ON    |
      | master_verify_checksum    | OFF   |
      | slave_sql_verify_checksum | ON    |
      +---------------------------+-------+
      6 rows in set (0.01 sec)
      

      Further debug info below:

      # resolve_stack_dump -s /tmp/mysqld.sym -n a
      0xedf27c my_print_stacktrace + 44
      0xd206f1 handle_fatal_signal + 1121
      0x3f6e80f7e0 _end + 1822088024
      0x3f6e032495 _end + 1813841933
      0x3f6e033c75 _end + 1813848045
      0x777028 _ZStlsISt11char_traitsIcEERSt13basic_ostreamIcT_ES5_PKc.part.1 + 0
      0x103defa _ZL27log_online_is_valid_log_segPKhm + 522
      0x1041090 _Z26log_online_follow_redo_logv + 1024
      0x111de0d srv_redo_log_follow_thread + 189
      0x3f6e807aa1 _end + 1822055961
      0x3f6e0e8bcd _end + 1814589253
      
      # addr2line -fie /usr/lib/debug/usr/sbin/mysqld.debug 0x777028
      operator<< <std::char_traits<char> >
      /opt/percona-devtoolset/root/usr/include/c++/4.8.2/ostream:530
      
      # addr2line -fie /usr/lib/debug/usr/sbin/mysqld.debug 0x103defa
      log_online_is_valid_log_seg
      /usr/src/debug/percona-server-5.7.24-27/percona-server-5.7.24-27/storage/innobase/log/log0online.cc:946
      
      # addr2line -fie /usr/lib/debug/usr/sbin/mysqld.debug 0x1041090
      log_online_follow_log_seg
      /usr/src/debug/percona-server-5.7.24-27/percona-server-5.7.24-27/storage/innobase/log/log0online.cc:1041
      log_online_follow_log_group
      /usr/src/debug/percona-server-5.7.24-27/percona-server-5.7.24-27/storage/innobase/log/log0online.cc:1099
      _Z26log_online_follow_redo_logv
      /usr/src/debug/percona-server-5.7.24-27/percona-server-5.7.24-27/storage/innobase/log/log0online.cc:1304
      
      # addr2line -fie /usr/lib/debug/usr/sbin/mysqld.debug 0x111de0d
      srv_redo_log_follow_thread
      /usr/src/debug/percona-server-5.7.24-27/percona-server-5.7.24-27/storage/innobase/srv/srv0srv.cc:2245
      

      Digging through the code, seems the log_block_get_checksum() and log_block_calc_checksum() functions are returning non-matching values in log0recv.cc#L661 hence causing the assert

      Attachments

        Activity

          People

            lalit.choudhary Lalit Choudhary
            Groenewold Ivan
            Votes:
            1 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Smart Checklist