Uploaded image for project: 'Percona Server for MySQL'
  1. Percona Server for MySQL
  2. PS-5676

Ocassional crashes due to Log block checksum mismatch when innodb_track_changed_pages=1


    • Bug
    • Status: Done
    • Medium
    • Resolution: Incomplete
    • 5.7.24-26
    • None
    • None


      Seen the following crash on multiple servers that were upgraded from 5.6 and are now running 5.7.24 with innodb_track_changed_pages=1.

      2019-06-05T12:50:02.122869Z 0 [ERROR] InnoDB: Log block checksum mismatch: LSN 33890770911232, expected 2870857039, calculated checksum 969592942, stored log block n:o 694660673, expected log block n:o 694660673
      2019-06-05 05:50:02 0x7f63e77f7700  InnoDB: Assertion failure in thread 140067062380288 in file log0online.cc line 946
      InnoDB: We intentionally generate a memory trap.
      InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
      InnoDB: If you get repeated assertion failures or crashes, even
      InnoDB: immediately after the mysqld startup, there may be
      InnoDB: corruption in the InnoDB tablespace. Please refer to
      InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
      InnoDB: about forcing recovery.
      12:50:02 UTC - mysqld got signal 6 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
      Attempting to collect some information that could help diagnose the problem.
      As this is a crash and something is definitely wrong, the information
      collection process might fail.
      Please help us make Percona Server better by reporting any
      bugs at http://bugs.percona.com/
      It is possible that mysqld could use up to
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 1174370 K  bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.
      Thread pointer: 0x0
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0 thread_stack 0x40000
      You may download the Percona Server operations manual by visiting
      http://www.percona.com/software/percona-server/. You may find information
      in the manual which will help you identify the cause of the crash.

      Checksum format being used is crc32:

      Welcome to the MySQL monitor.  Commands end with ; or \g.
      Your MySQL connection id is 8627
      Server version: 5.7.24-27-log Percona Server (GPL), Release 27, Revision bd42700
      Copyright (c) 2009-2018 Percona LLC and/or its affiliates
      Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.
      Oracle is a registered trademark of Oracle Corporation and/or its
      affiliates. Other names may be trademarks of their respective
      Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
      mysql> show variables like '%checksum%';
      | Variable_name             | Value |
      | binlog_checksum           | CRC32 |
      | innodb_checksum_algorithm | crc32 |
      | innodb_checksums          | ON    |
      | innodb_log_checksums      | ON    |
      | master_verify_checksum    | OFF   |
      | slave_sql_verify_checksum | ON    |
      6 rows in set (0.01 sec)

      Further debug info below:

      # resolve_stack_dump -s /tmp/mysqld.sym -n a
      0xedf27c my_print_stacktrace + 44
      0xd206f1 handle_fatal_signal + 1121
      0x3f6e80f7e0 _end + 1822088024
      0x3f6e032495 _end + 1813841933
      0x3f6e033c75 _end + 1813848045
      0x777028 _ZStlsISt11char_traitsIcEERSt13basic_ostreamIcT_ES5_PKc.part.1 + 0
      0x103defa _ZL27log_online_is_valid_log_segPKhm + 522
      0x1041090 _Z26log_online_follow_redo_logv + 1024
      0x111de0d srv_redo_log_follow_thread + 189
      0x3f6e807aa1 _end + 1822055961
      0x3f6e0e8bcd _end + 1814589253
      # addr2line -fie /usr/lib/debug/usr/sbin/mysqld.debug 0x777028
      operator<< <std::char_traits<char> >
      # addr2line -fie /usr/lib/debug/usr/sbin/mysqld.debug 0x103defa
      # addr2line -fie /usr/lib/debug/usr/sbin/mysqld.debug 0x1041090
      # addr2line -fie /usr/lib/debug/usr/sbin/mysqld.debug 0x111de0d

      Digging through the code, seems the log_block_get_checksum() and log_block_calc_checksum() functions are returning non-matching values in log0recv.cc#L661 hence causing the assert




            lalit.choudhary Lalit Choudhary
            Groenewold Ivan
            1 Vote for this issue
            7 Start watching this issue



              Smart Checklist