Uploaded image for project: 'Percona Server for MySQL'
  1. Percona Server for MySQL
  2. PS-5367

Crash Recovery doesn't work in Percona Server with MyRocks


    • Type: Bug
    • Status: On Hold
    • Priority: Medium
    • Resolution: Reopened
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Documentation, MyRocks
    • Labels:


      I am testing against Percona Server 5.7.22-22. All tables except for system tables have RocksDB storage engine.


      mysql> select distinct table_schema from information_schema.tables where engine != 'ROCKSDB'; 
      | table_schema | 
      | information_schema | 
      | mysql | 
      | performance_schema | 
      | sys | 
      4 rows in set (2.55 sec)



      The test case is simple: 

      • Startup MySQL under docker 
      • Run some load against RocksDB tables 
      • While load is running stop the container (which effectively kills mysqld)

      MySQL fails to come up as it complains about prepared transactions. 

      For reference here is the log from mysqld startup after unclean shutdown: 

      {"log":"2019-01-18T19:19:41.421422Z 0 [Note] Plugin 'FEDERATED' is disabled.\n","stream":"stderr","time":"2019-01-18T19:19:41.421571232Z"} 
      {"log":"2019-01-18T19:19:41.423849Z 0 [Note] InnoDB: Buffer pool(s) load completed at 190118 19:19:41\n","stream":"stderr","time":"2019-01-18T19:19:41.424039477Z"} 
      {"log":"2019-01-18T19:19:41.436563Z 0 [Note] RocksDB: 2 column families found\n","stream":"stderr","time":"2019-01-18T19:19:41.436771081Z"} 
      {"log":"2019-01-18T19:19:41.436687Z 0 [Note] RocksDB: Column Families at start:\n","stream":"stderr","time":"2019-01-18T19:19:41.436781766Z"} 
      {"log":"2019-01-18T19:19:41.436722Z 0 [Note] cf=default\n","stream":"stderr","time":"2019-01-18T19:19:41.436785738Z"} 
      {"log":"2019-01-18T19:19:41.436727Z 0 [Note] write_buffer_size=67108864\n","stream":"stderr","time":"2019-01-18T19:19:41.436789208Z"} 
      {"log":"2019-01-18T19:19:41.436729Z 0 [Note] target_file_size_base=33554432\n","stream":"stderr","time":"2019-01-18T19:19:41.436792448Z"} 
      {"log":"2019-01-18T19:19:41.436753Z 0 [Note] cf=__system__\n","stream":"stderr","time":"2019-01-18T19:19:41.43679574Z"} 
      {"log":"2019-01-18T19:19:41.436757Z 0 [Note] write_buffer_size=67108864\n","stream":"stderr","time":"2019-01-18T19:19:41.436799008Z"} 
      {"log":"2019-01-18T19:19:41.436758Z 0 [Note] target_file_size_base=33554432\n","stream":"stderr","time":"2019-01-18T19:19:41.436802746Z"} 
      {"log":"2019-01-18T19:19:43.123242Z 0 [Note] RocksDB: Table_store: loaded DDL data for 36872 tables\n","stream":"stderr","time":"2019-01-18T19:19:43.1235494Z"} 
      {"log":"2019-01-18T19:19:43.128706Z 0 [Note] RocksDB instance opened\n","stream":"stderr","time":"2019-01-18T19:19:43.128885179Z"} 
      {"log":"2019-01-18T19:19:43.129060Z 0 [Note] Found 3 prepared transaction(s) in ROCKSDB\n","stream":"stderr","time":"2019-01-18T19:19:43.129466557Z"} 
      {"log":"2019-01-18T19:19:43.129081Z 0 [ERROR] Found 3 prepared transactions! It means that mysqld was not shut down properly last time and critical recovery information (last binlog or tc.log file) was manually deleted after a crash. You have to start mysqld with --tc-heuristic-recover switch to commit or rollback pending transactions.\n","stream":"stderr","time":"2019-01-18T19:19:43.12948232Z"} 
      {"log":"2019-01-18T19:19:43.129088Z 0 [ERROR] Aborting\n","stream":"stderr","time":"2019-01-18T19:19:43.129491008Z"} 
      {"log":"2019-01-18T19:19:43.129104Z 0 [Note] Binlog end\n","stream":"stderr","time":"2019-01-18T19:19:43.129498713Z"}

      Looking at PS 5.7 code it seems like it will never handle a case cleanly when there are two transactional engines. Check out https://github.com/percona/percona-server/blob/5.7/sql/xa.cc#L161-L180total_ha_2pc

      Because total_ha_2pc=3 the info.dry_run variable would be set to "true". Percona Server with MyRocks engine has 2 transactional engines enabled `InnoDB` and `RocksDB`, hence recovery of prepared statements will never be performed by default.

      Thanks to Yoshinori for pointing out the above.

        Smart Checklist




              • Assignee:
                patrick.birch Patrick Birch
                ot Ovais Tariq
              • Votes:
                1 Vote for this issue
                6 Start watching this issue


                • Created:

                  Time Tracking

                  Original Estimate - Not Specified
                  Not Specified
                  Remaining Estimate - Not Specified
                  Not Specified
                  Time Spent - 10 minutes