Uploaded image for project: 'Percona Server'
  1. Percona Server
  2. PS-5366

RocksDB: crash recovery not working with Operation timed out: Timeout waiting to lock key

    Details

      Description

      RocksDB crash recovery is not working with same message as in https://jira.percona.com/browse/PS-4510

      2019-01-19T05:44:27.850208Z 0 [Note] RocksDB: 2 column families found
      2019-01-19T05:44:27.850325Z 0 [Note] RocksDB: Column Families at start:
      2019-01-19T05:44:27.850350Z 0 [Note] cf=default
      2019-01-19T05:44:27.850358Z 0 [Note] write_buffer_size=67108864
      2019-01-19T05:44:27.850363Z 0 [Note] target_file_size_base=67108864
      2019-01-19T05:44:27.850385Z 0 [Note] cf=_system_
      2019-01-19T05:44:27.850391Z 0 [Note] write_buffer_size=67108864
      2019-01-19T05:44:27.850395Z 0 [Note] target_file_size_base=67108864
      2019-01-19T05:44:28.097723Z 0 [ERROR] RocksDB: Error opening instance, Status Code: 9, Status: Operation timed out: Timeout waiting to lock key
      2019-01-19T05:44:28.097755Z 0 [ERROR] Plugin 'ROCKSDB' init function returned error.
      2019-01-19T05:44:28.097762Z 0 [ERROR] Plugin 'ROCKSDB' registration as a STORAGE ENGINE failed.

       

      You can use the script:

       

      #!/bin/bash
      NAME=${USER}.p57.rocksdb.recovery
      docker run -d -p 3306:3306 --name $NAME -e MYSQL_ROOT_PASSWORD=secret percona:5.7.24 --server-id=51 --log-bin=mysqld-bin --loose-rocksdb_wal_recovery_mode=3
      docker exec -i $NAME bash -c "while ! (mysqladmin --protocol=tcp --host=127.0.0.1 -uroot -psecret --silent -c 3 -i 5 --wait=300 ping &>/dev/null) ;do echo -n . ; sleep 5 ; done"
      docker exec -i $NAME ps-admin --user=root --password=secret --enable-rocksdb
      docker exec -i $NAME mysql --user=root --password=secret -e 'select version();'
      docker exec -i $NAME mysql --user=root --password=secret -e 'create database test; create table test.t (id int auto_increment primary key, pad char(200)) engine=ROCKSDB;'
      docker exec $NAME mysqlslap -uroot -psecret -h 127.0.0.1 --protocol=tcp -q 'INSERT INTO test.t(pad) values("a");' --create-schema=performance_schema --number-of-queries=1000000 -c 10 &
      sleep 5
      sudo kill -9 $(pgrep -xn mysqld)
      sleep 5
      docker start $NAME
      sleep 15
      docker logs $NAME
      docker rm -f $NAME
      

       

      As you can see there is a single table, insert to rocksdb, all default options and classic crash recovery case after kill -9 is not working at all.

       

        Smart Checklist

          Attachments

            Activity

              People

              • Assignee:
                george.lorch George Lorch
                Reporter:
                nickolay.ihalainen Nickolay Ihalainen
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - Not Specified
                  Not Specified
                  Logged:
                  Time Spent - 4 hours
                  4h