Uploaded image for project: 'Percona Server for MySQL'
  1. Percona Server for MySQL
  2. PS-8257

Failed upgrade attempt to 8.0.29 corrupts the data dictionary

Details

    • Bug
    • Status: Done
    • High
    • Resolution: Fixed
    • 8.0.29-21 (Q2 2022)
    • 8.0.29-21 (Q2 2022)
    • None
    • None
    • Yes

    Description

      When an upgrade from 8.0.27 to 8.0.29 is not possible the data dictionary gets corrupted and 8.0.27 does not start with theused data_dir.

      To reproduce the issue:

      1. Start a 8.0.27 instance.

      2. Modify 8.0.29 so the upgrade fails by introducing the following patch:

      ```
      diff --git a/sql/dd/impl/upgrade/server.cc b/sql/dd/impl/upgrade/server.cc
      index 05cc418316f..58a97e9b038 100644
      — a/sql/dd/impl/upgrade/server.cc
      +++ b/sql/dd/impl/upgrade/server.cc
      @@ -554,6 +554,7 @@ static bool get_shared_tablespace_names(
       static bool check_tables(THD *thd, std::unique_ptr<Schema> &schema,
                                const std::set<dd::String_type> *shared_spaces,
                                Upgrade_error_counter *error_count)

      { +  fprintf(stderr, "TODOLUIS - %s\n", __PRETTY_FUNCTION__);    std::unique_ptr<Object_key> table_key(        dd::Table::DD_table::create_key_by_schema_id(schema->id())); @@ -583,6 +584,10 @@ static bool check_tables(THD *thd, std::unique_ptr<Schema> &schema,          }

             }
           }
      +
      +    (*error_count)++;  // Force upgrade error
      +
           return error_count->has_too_many_errors();
         };
      ```

      3. Start 8.0.29 with the `data_dir` of 8.0.27. The start fails with:

      ```
      2022-05-25T10:06:25.650972Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
      2022-05-25T10:06:25.651198Z 0 [ERROR] [MY-010119] [Server] Aborting
      ```

      4. Start 8.0.27 with the same `data_dir`. The start fails with:

      ```
      2022-05-25T10:07:09.582930Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
      2022-05-25T10:07:09.752854Z 1 [ERROR] [MY-013183] [InnoDB] Assertion failure: mtr0log.cc:135:*type <= MLOG_BIGGEST_TYPE thread 139626150885120
      InnoDB: We intentionally generate a memory trap.
      InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
      InnoDB: If you get repeated assertion failures or crashes, even
      InnoDB: immediately after the mysqld startup, there may be
      InnoDB: corruption in the InnoDB tablespace. Please refer to
      InnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html
      InnoDB: about forcing recovery.
      10:07:09 UTC - mysqld got signal 6 ;
      Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
      Thread pointer: 0x7efd538f3000
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 7efd3f1fd448 thread_stack 0x100000
      /home/ldonoso/src/mysql-8.0.27-bld/install/bin/mysqld(my_print_stacktrace(unsigned char const*, unsigned long)+0x42) [0x62e2502]
      /home/ldonoso/src/mysql-8.0.27-bld/install/bin/mysqld(handle_fatal_signal+0x232) [0x5176322]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7efd567f7420]
      /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb) [0x7efd53dd200b]
      /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b) [0x7efd53db1859]
      /home/ldonoso/src/mysql-8.0.27-bld/install/bin/mysqld(ut_dbg_assertion_failed(char const*, char const*, unsigned long)+0x293) [0x6aa0763]
      /home/ldonoso/src/mysql-8.0.27-bld/install/bin/mysqld(mlog_parse_initial_log_record(unsigned char const*, unsigned char const*, mlog_id_t*, unsigned int*, unsigned int*)+0x89) [0x686fa89]
      /home/ldonoso/src/mysql-8.0.27-bld/install/bin/mysqld() [0x684b952]
      /home/ldonoso/src/mysql-8.0.27-bld/install/bin/mysqld() [0x684af32]
      /home/ldonoso/src/mysql-8.0.27-bld/install/bin/mysqld() [0x68498a1]
      /home/ldonoso/src/mysql-8.0.27-bld/install/bin/mysqld() [0x68490d9]
      /home/ldonoso/src/mysql-8.0.27-bld/install/bin/mysqld() [0x684280d]
      /home/ldonoso/src/mysql-8.0.27-bld/install/bin/mysqld(recv_recovery_from_checkpoint_start(log_t&, unsigned long)+0x71b) [0x684136b]
      /home/ldonoso/src/mysql-8.0.27-bld/install/bin/mysqld(srv_start(bool)+0x2733) [0x69e8143]
      /home/ldonoso/src/mysql-8.0.27-bld/install/bin/mysqld() [0x637d9e1]
      /home/ldonoso/src/mysql-8.0.27-bld/install/bin/mysqld() [0x636da81]
      /home/ldonoso/src/mysql-8.0.27-bld/install/bin/mysqld(dd::bootstrap::DDSE_dict_init(THD*, dict_init_mode_t, unsigned int)+0x66) [0x5f90176]
      /home/ldonoso/src/mysql-8.0.27-bld/install/bin/mysqld(dd::upgrade_57::do_pre_checks_and_initialize_dd(THD*)+0x87c) [0x6291c0c]
      /home/ldonoso/src/mysql-8.0.27-bld/install/bin/mysqld() [0x4b4e05a]
      /home/ldonoso/src/mysql-8.0.27-bld/install/bin/mysqld() [0x6b8fae0]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7efd567eb609]
      /lib/x86_64-linux-gnu/libc.so.6(clone+0x43) [0x7efd53eae133]

      Trying to get some variables.
      Some pointers may be invalid and cause the dump to abort.
      Query (0): Connection ID (thread ID): 1
      Status: NOT_KILLED

      The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
      information that should help you find out what is causing the crash.
      Writing a core file
      Aborted (core dumped)
      `

      Notes:

      • The data_dir is not corrupted when repeating the steps from 8.0.27 to .28

      Attachments

        Activity

          People

            luis.donoso Luis Donoso (Inactive)
            luis.donoso Luis Donoso (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - Not Specified
                Not Specified
                Logged:
                Time Spent - 2 days
                2d

                Smart Checklist