1. Add new MyRocks Information Schema Table `ROCKSDB_LIVE_FILES_METADATA` to https://www.percona.com/doc/percona-server/LATEST/myrocks/information-schema-tables.html
2. Removed variables:
"rocksdb_instant_ddl" was replaced with "rocksdb_disable_instant_ddl".
3. New variables:
a) Add a new variable "rocksdb_alter_table_comment_inplace":
b) Add a new variable "rocksdb_bulk_load_fail_if_not_bottommost_level":
c) Add a new variable "rocksdb_charge_memory":
d) Add a new variable "rocksdb_column_default_value_as_expression":
e) Add a new variable "rocksdb_corrupt_data_action":
f) Add a new variable "rocksdb_disable_instant_ddl":
g) Add a new variable "rocksdb_enable_delete_range_for_drop_index":
h) Add a new variable "rocksdb_partial_index_blind_delete":
i) Add a new variable "rocksdb_protection_bytes_per_key":
j) Add a new variable "rocksdb_use_write_buffer_manager":
Commit descriptions that may help with describing new variables:
1. Add INFORMATION_SCHEMA.ROCKSDB_LIVE_FILES_METADATA table
Add a new table `INFORMATION_SCHEMA.ROCKSDB_LIVE_FILES_METADATA`,
which lists the live files metadata information from rocksdb API
2. support change myrocks table comment inplace - rocksdb_alter_table_comment_inplace
Currently inplace change table comment for myrocks table isn't support. thus it will take long time to change table comment.
This change is to add inplace support for some change table comment scenario .
Inplace change table comment scenarios:
1. change ttl duration value, or
2. add random string into table comments.
1. add a variable rocksdb_alter_table_comment_inplace to enable or disable this functional. by default, it is disabled
2. During DDL check_if_supported_inplace_alter, check whether any new ttl col is added(removed) or any new ttl duration is added. only ttl duration change and random string change support inplaces.
3. During DDL prepare_inplace_alter_table, update these keydef to use these new ttl duration value.
4. During DDL commit_inplace_alter_table, commit all changes together
3. Use RocksDB DeleteRange for dropping indexes - rocksdb_enable_delete_range_for_drop_index
Prior to this diff we issued a CompactRange() and relied on the compaction filter to drop the keys remaining after DeleteFileInRange() for drop index or drop table. Now we have used delete range which is instantaneous.
- New drop table or drop index will directly call the DeleteRange in sync and avoid triggering the background worker thread. All the dictionary updates are kept same as previous behavior. Note that currently DeleteRange is not transactional, so write batch is used. I kept the write batch for DeleteRange and dictionary update separate as of now.
- Added a variable rocksdb_enable_delete_range_for_drop_index to control the old vs new behavior.
- Updated the combination file to test the new behavior.
- Once we have rolled out these changes, we will drop the background used for drop index.
4. add DD data during instant ddl - rocksdb_disable_instant_ddl
Instant DDL for myrocks requires to store extra data into Data Dictionary(DD)
- mysql.columns: store 'original' default value for instant columns into se_private field and used during read path.
- DD_INSTANT_COLUMN_DEFAULT_NULL: this is a null value for column, such as default_null=1
- DD_INSTANT_COLUMN_DEFAULT: this is non-null value for column, such as default=0x1234
- mysql.tables: store 'original' number of non-instant columns before first instant column. such as instant_col=2. This data is an optimization and maybe removed later after adding record version support.
- rocksdb-disable-instant-ddl: change variable name from rocksdb-instant-ddl to rocksdb-disable-instant-ddl. it will be a gatekeep for myrocks instant ddl.
- add instant DDL MTR to verify DD data is correct.
- add sql_dd.cc/sql_dd.h files: These files will be used to read/write data from sql DD.
5. Enable RocksDB memory-charging features configuration from cnf - rocksdb_charge_memory, rocksdb_use_write_buffer_manager
Summary: Enable MyRocks to turn on RocksDB memory-charging related features (i.e, BlockBasedTableOptions::cache_usage_options.options.charged; WriteBufferManager) from cnf files for feature evaluation
6. add rocksdb_bulk_load_fail_if_not_bottommost_level option
Summary: - create variable `rocksdb_bulk_load_fail_if_not_bottommost_level`, by default it is OFF, when turns on bulk load will fail when the sst files cannot be placed to the bottommost level in rocksdb
7. Add option for KV checksum in WriteBatch - rocksdb_protection_bytes_per_key
Added the session variable, `protection_bytes_per_key`, that can be used to configure `WriteOptions::protection_bytes_per_key`. By default it is 0 (disabled). When set to 1, 2, 4, or 8, it uses that number of bytes per KV to protect entries in the WriteBatch.
8. Add flag to check for materialized groups before issuing SD - rocksdb_partial_index_blind_delete
Today, when partial indexes need maintenance, an SD is always issued
blindly in order to save a read. This could potentially cause tombstones to build up
though, and the cost of performing a point lookup turns out to be not
Add a flag to control this behaviour so that we can adjust if this becomes
a problem later.
9. Add an option to abort/ignore server when hitting data corruption - rocksdb_corrupt_data_action
When hitting the HA_ERR_ROCKSDB_CORRUPT_DATA error in myrocks, the current behavior is to fail the query. We want to change the behavior to provide a choice between:
1. Fail the query with error HA_ERR_ROCKSDB_CORRUPT_DATA
2. Crash the server
3. Pass the query with warning
Introduce rocksdb_corrupt_data_action flag to control this behaviour.
10. Add rocksdb_column_default_value_as_expression and more testing
Summary: add rocksdb_column_default_value_as_expressoin variable to control whether support column default value as expression/function and also add more testing