Uploaded image for project: 'Percona Server for MySQL'
  1. Percona Server for MySQL
  2. PS-8653

Document new features in percona-202203 to percona-202206

Details

    • New Feature
    • Status: In Progress
    • Medium
    • Resolution: Unresolved
    • 8.0.x, Not 5.6.x, Not 5.7.x
    • None
    • Documentation, MyRocks
    • None
    • Yes

    Description

      https://github.com/percona/percona-server/pull/5031

      1. Add new MyRocks Information Schema Table `ROCKSDB_LIVE_FILES_METADATA` to https://www.percona.com/doc/percona-server/LATEST/myrocks/information-schema-tables.html

      CREATE TABLE `ROCKSDB_LIVE_FILES_METADATA` (
        `CF_NAME` varchar(193) NOT NULL DEFAULT '',
        `LEVEL` varchar(513) NOT NULL DEFAULT '',
        `NAME` varchar(513) NOT NULL DEFAULT '',
        `DB_PATH` varchar(513) NOT NULL DEFAULT '',
        `FILE_NUMBER` bigint NOT NULL DEFAULT '0',
        `FILE_TYPE` varchar(193) NOT NULL DEFAULT '',
        `SIZE` bigint NOT NULL DEFAULT '0',
        `RELATIVE_FILENAME` varchar(193) NOT NULL DEFAULT '',
        `DIRECTORY` varchar(513) NOT NULL DEFAULT '',
        `TEMPERATURE` varchar(193) NOT NULL DEFAULT '',
        `FILE_CHECKSUM` varchar(513) NOT NULL DEFAULT '',
        `FILE_CHECKSUM_FUNC_NAME` varchar(193) NOT NULL DEFAULT '',
        `SMALLEST_SEQNO` bigint NOT NULL DEFAULT '0',
        `LARGEST_SEQNO` bigint NOT NULL DEFAULT '0',
        `SMALLEST_KEY` varchar(513) NOT NULL DEFAULT '',
        `LARGEST_KEY` varchar(513) NOT NULL DEFAULT '',
        `NUM_READS_SAMPLED` bigint NOT NULL DEFAULT '0',
        `BEING_COMPACTED` tinyint NOT NULL DEFAULT '0',
        `NUM_ENTRIES` bigint NOT NULL DEFAULT '0',
        `NUM_DELETIONS` bigint NOT NULL DEFAULT '0',
        `OLDEST_BLOB_FILE_NUMBER` bigint NOT NULL DEFAULT '0',
        `OLDEST_ANCESTER_TIME` bigint NOT NULL DEFAULT '0',
        `FILE_CREATION_TIME` bigint NOT NULL DEFAULT '0'
      )
      

      2. Removed variables:
      "rocksdb_instant_ddl" was replaced with "rocksdb_disable_instant_ddl".

      3. New variables:
      a) Add a new variable "rocksdb_alter_table_comment_inplace":

         :dyn: Yes
         :scope: Global
         :vartype: Bool
         :default: OFF
      "Allow inplace alter for alter table comment."
      

      b) Add a new variable "rocksdb_bulk_load_fail_if_not_bottommost_level":

         :dyn: Yes
         :scope: Global, Session
         :vartype: Bool
         :default: OFF
          "Fail the bulk load if a sst file created from bulk load "
          "cannot fit in the rocksdb bottommost level. "
          "Turn off this variable could have severe performance impact. "
          "Can be changed only when bulk load is disabled.",
      

      c) Add a new variable "rocksdb_charge_memory":

         :dyn: No
         :scope: Global
         :vartype: Bool
         :default: OFF
           "For experiment only. Turn on memory "
           "charging feature of RocksDB",
      

      d) Add a new variable "rocksdb_column_default_value_as_expression":

         :dyn: Yes
         :scope: Global
         :vartype: Bool
         :default: ON
         "allow column default value expressed in function",
      

      e) Add a new variable "rocksdb_corrupt_data_action":

         :dyn: Yes
         :scope: Global
         :vartype: enum { ERROR = 0, ABORT_SERVER, WARNING };
         :default: ERROR
          "Control behavior when hitting data corruption. We can fail the query, "
          "crash the server or pass the query and give users a warning. ",
      

      f) Add a new variable "rocksdb_disable_instant_ddl":

         :dyn: Yes
         :scope: Global
         :vartype: Bool
         :default: ON
          "Disable instant ddl during alter table"
      

      g) Add a new variable "rocksdb_enable_delete_range_for_drop_index":

         :dyn: Yes
         :scope: Global
         :vartype: Bool
         :default: OFF
          "Enable drop table/index by delete range"
      

      h) Add a new variable "rocksdb_partial_index_blind_delete":

         :dyn: Yes
         :scope: Global
         :vartype: Bool
         :default: ON
          "If OFF, always read from partial index to check if key exists before "
          "deleting. Otherwise, delete marker is unconditionally written."
      

      i) Add a new variable "rocksdb_protection_bytes_per_key":

         :dyn: Yes
         :scope: Global, Session
         :vartype: Numeric
         :default: 0
         :min: 0
         :max: ULONG_MAX (0xFFFFFFFF)
        "WriteOptions::protection_bytes_per_key for RocksDB"
      

      j) Add a new variable "rocksdb_use_write_buffer_manager":

         :dyn: No
         :scope: Global
         :vartype: Bool
         :default: OFF
         "For experiment only. Use write buffer manager",
      

      Commit descriptions that may help with describing new variables:

      1. Add INFORMATION_SCHEMA.ROCKSDB_LIVE_FILES_METADATA table
      Summary:
      Add a new table `INFORMATION_SCHEMA.ROCKSDB_LIVE_FILES_METADATA`,
      which lists the live files metadata information from rocksdb API
      `GetLiveFilesMetaData()`.

      2. support change myrocks table comment inplace - rocksdb_alter_table_comment_inplace
      Summary:
      Currently inplace change table comment for myrocks table isn't support. thus it will take long time to change table comment.
      This change is to add inplace support for some change table comment scenario .
      Inplace change table comment scenarios:
      1. change ttl duration value, or
      2. add random string into table comments.
      Details:
      1. add a variable rocksdb_alter_table_comment_inplace to enable or disable this functional. by default, it is disabled
      2. During DDL check_if_supported_inplace_alter, check whether any new ttl col is added(removed) or any new ttl duration is added. only ttl duration change and random string change support inplaces.
      3. During DDL prepare_inplace_alter_table, update these keydef to use these new ttl duration value.
      4. During DDL commit_inplace_alter_table, commit all changes together

      3. Use RocksDB DeleteRange for dropping indexes - rocksdb_enable_delete_range_for_drop_index
      Summary:
      Prior to this diff we issued a CompactRange() and relied on the compaction filter to drop the keys remaining after DeleteFileInRange() for drop index or drop table. Now we have used delete range which is instantaneous.
      Changes

      • New drop table or drop index will directly call the DeleteRange in sync and avoid triggering the background worker thread. All the dictionary updates are kept same as previous behavior. Note that currently DeleteRange is not transactional, so write batch is used. I kept the write batch for DeleteRange and dictionary update separate as of now.
      • Added a variable rocksdb_enable_delete_range_for_drop_index to control the old vs new behavior.
      • Updated the combination file to test the new behavior.
        *Next changes*
      • Once we have rolled out these changes, we will drop the background used for drop index.

      4. add DD data during instant ddl - rocksdb_disable_instant_ddl
      Summary:
      Instant DDL for myrocks requires to store extra data into Data Dictionary(DD)

      • mysql.columns: store 'original' default value for instant columns into se_private field and used during read path.
      • DD_INSTANT_COLUMN_DEFAULT_NULL: this is a null value for column, such as default_null=1
      • DD_INSTANT_COLUMN_DEFAULT: this is non-null value for column, such as default=0x1234
      • mysql.tables: store 'original' number of non-instant columns before first instant column. such as instant_col=2. This data is an optimization and maybe removed later after adding record version support.
        Details:
      • rocksdb-disable-instant-ddl: change variable name from rocksdb-instant-ddl to rocksdb-disable-instant-ddl. it will be a gatekeep for myrocks instant ddl.
      • add instant DDL MTR to verify DD data is correct.
      • add sql_dd.cc/sql_dd.h files: These files will be used to read/write data from sql DD.

      5. Enable RocksDB memory-charging features configuration from cnf - rocksdb_charge_memory, rocksdb_use_write_buffer_manager
      Summary: Enable MyRocks to turn on RocksDB memory-charging related features (i.e, BlockBasedTableOptions::cache_usage_options.options.charged; WriteBufferManager) from cnf files for feature evaluation

      6. add rocksdb_bulk_load_fail_if_not_bottommost_level option
      Summary: - create variable `rocksdb_bulk_load_fail_if_not_bottommost_level`, by default it is OFF, when turns on bulk load will fail when the sst files cannot be placed to the bottommost level in rocksdb

      7. Add option for KV checksum in WriteBatch - rocksdb_protection_bytes_per_key
      Summary:
      Added the session variable, `protection_bytes_per_key`, that can be used to configure `WriteOptions::protection_bytes_per_key`. By default it is 0 (disabled). When set to 1, 2, 4, or 8, it uses that number of bytes per KV to protect entries in the WriteBatch.

      8. Add flag to check for materialized groups before issuing SD - rocksdb_partial_index_blind_delete
      Summary:
      Today, when partial indexes need maintenance, an SD is always issued
      blindly in order to save a read. This could potentially cause tombstones to build up
      though, and the cost of performing a point lookup turns out to be not
      that bad.
      Add a flag to control this behaviour so that we can adjust if this becomes
      a problem later.

      9. Add an option to abort/ignore server when hitting data corruption - rocksdb_corrupt_data_action
      Summary:
      When hitting the HA_ERR_ROCKSDB_CORRUPT_DATA error in myrocks, the current behavior is to fail the query. We want to change the behavior to provide a choice between:
      1. Fail the query with error HA_ERR_ROCKSDB_CORRUPT_DATA
      2. Crash the server
      3. Pass the query with warning
      Introduce rocksdb_corrupt_data_action flag to control this behaviour.

      10. Add rocksdb_column_default_value_as_expression and more testing
      Summary: add rocksdb_column_default_value_as_expressoin variable to control whether support column default value as expression/function and also add more testing

      Attachments

        Activity

          People

            alina.derkach Alina Derkach
            przemyslaw.skibinski Przemyslaw Skibinski
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:

              Smart Checklist