Uploaded image for project: 'Percona Server'
  1. Percona Server
  2. PS-5367

Crash Recovery doesn't work in Percona Server with MyRocks

    Details

    • Type: Bug
    • Status: On Hold
    • Priority: Medium
    • Resolution: Reopened
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Documentation, MyRocks
    • Labels:
      None

      Description

      I am testing against Percona Server 5.7.22-22. All tables except for system tables have RocksDB storage engine.

       

      mysql> select distinct table_schema from information_schema.tables where engine != 'ROCKSDB'; 
      +--------------------+ 
      | table_schema | 
      +--------------------+ 
      | information_schema | 
      | mysql | 
      | performance_schema | 
      | sys | 
      +--------------------+ 
      4 rows in set (2.55 sec)
      

       

       

      The test case is simple: 

      • Startup MySQL under docker 
      • Run some load against RocksDB tables 
      • While load is running stop the container (which effectively kills mysqld)

      MySQL fails to come up as it complains about prepared transactions. 

      For reference here is the log from mysqld startup after unclean shutdown: 

      {"log":"2019-01-18T19:19:41.421422Z 0 [Note] Plugin 'FEDERATED' is disabled.\n","stream":"stderr","time":"2019-01-18T19:19:41.421571232Z"} 
      {"log":"2019-01-18T19:19:41.423849Z 0 [Note] InnoDB: Buffer pool(s) load completed at 190118 19:19:41\n","stream":"stderr","time":"2019-01-18T19:19:41.424039477Z"} 
      {"log":"2019-01-18T19:19:41.436563Z 0 [Note] RocksDB: 2 column families found\n","stream":"stderr","time":"2019-01-18T19:19:41.436771081Z"} 
      {"log":"2019-01-18T19:19:41.436687Z 0 [Note] RocksDB: Column Families at start:\n","stream":"stderr","time":"2019-01-18T19:19:41.436781766Z"} 
      {"log":"2019-01-18T19:19:41.436722Z 0 [Note] cf=default\n","stream":"stderr","time":"2019-01-18T19:19:41.436785738Z"} 
      {"log":"2019-01-18T19:19:41.436727Z 0 [Note] write_buffer_size=67108864\n","stream":"stderr","time":"2019-01-18T19:19:41.436789208Z"} 
      {"log":"2019-01-18T19:19:41.436729Z 0 [Note] target_file_size_base=33554432\n","stream":"stderr","time":"2019-01-18T19:19:41.436792448Z"} 
      {"log":"2019-01-18T19:19:41.436753Z 0 [Note] cf=__system__\n","stream":"stderr","time":"2019-01-18T19:19:41.43679574Z"} 
      {"log":"2019-01-18T19:19:41.436757Z 0 [Note] write_buffer_size=67108864\n","stream":"stderr","time":"2019-01-18T19:19:41.436799008Z"} 
      {"log":"2019-01-18T19:19:41.436758Z 0 [Note] target_file_size_base=33554432\n","stream":"stderr","time":"2019-01-18T19:19:41.436802746Z"} 
      {"log":"2019-01-18T19:19:43.123242Z 0 [Note] RocksDB: Table_store: loaded DDL data for 36872 tables\n","stream":"stderr","time":"2019-01-18T19:19:43.1235494Z"} 
      {"log":"2019-01-18T19:19:43.128706Z 0 [Note] RocksDB instance opened\n","stream":"stderr","time":"2019-01-18T19:19:43.128885179Z"} 
      {"log":"2019-01-18T19:19:43.129060Z 0 [Note] Found 3 prepared transaction(s) in ROCKSDB\n","stream":"stderr","time":"2019-01-18T19:19:43.129466557Z"} 
      {"log":"2019-01-18T19:19:43.129081Z 0 [ERROR] Found 3 prepared transactions! It means that mysqld was not shut down properly last time and critical recovery information (last binlog or tc.log file) was manually deleted after a crash. You have to start mysqld with --tc-heuristic-recover switch to commit or rollback pending transactions.\n","stream":"stderr","time":"2019-01-18T19:19:43.12948232Z"} 
      {"log":"2019-01-18T19:19:43.129088Z 0 [ERROR] Aborting\n","stream":"stderr","time":"2019-01-18T19:19:43.129491008Z"} 
      {"log":"\n","stream":"stderr","time":"2019-01-18T19:19:43.129495135Z"} 
      {"log":"2019-01-18T19:19:43.129104Z 0 [Note] Binlog end\n","stream":"stderr","time":"2019-01-18T19:19:43.129498713Z"}
      

      Looking at PS 5.7 code it seems like it will never handle a case cleanly when there are two transactional engines. Check out https://github.com/percona/percona-server/blob/5.7/sql/xa.cc#L161-L180total_ha_2pc

      Because total_ha_2pc=3 the info.dry_run variable would be set to "true". Percona Server with MyRocks engine has 2 transactional engines enabled `InnoDB` and `RocksDB`, hence recovery of prepared statements will never be performed by default.

      Thanks to Yoshinori for pointing out the above.

        Smart Checklist

          Attachments

            Activity

              People

              • Assignee:
                patrick.birch Patrick Birch
                Reporter:
                ot Ovais Tariq
              • Votes:
                1 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - Not Specified
                  Not Specified
                  Logged:
                  Time Spent - 10 minutes
                  10m