-
Type:
Bug
-
Status: Done
-
Priority:
Medium
-
Resolution: Invalid
-
Affects Version/s: 5.7.19-17
-
Fix Version/s: 5.7.20-18
-
Component/s: MyRocks
-
Labels:
-
Environment:
Percona Server 5.7.19-17
Copied settings from facebook wiki page on server with 20GB RAM and 10GB swap:
loose_rocksdb_max_open_files=-1
loose_rocksdb_max_background_jobs=8
loose_rocksdb_max_total_wal_size=4G
loose_rocksdb_block_size=16384
loose_rocksdb_block_cache_size=32G
loose_rocksdb_table_cache_numshardbits=6- rate limiter
loose_rocksdb_bytes_per_sync=4194304
loose_rocksdb_wal_bytes_per_sync=4194304
loose_rocksdb_rate_limiter_bytes_per_sec=104857600 #100MB/s. Increase if you're running on higher spec machines
- triggering compaction if there are many sequential deletes
loose_rocksdb_compaction_sequential_deletes_count_sd=1
loose_rocksdb_compaction_sequential_deletes=199999
loose_rocksdb_compaction_sequential_deletes_window=200000
- read free replication
loose_rocksdb_rpl_lookup_rows=0
loose_rocksdb_default_cf_options=write_buffer_size=128m;target_file_size_base=32m;max_bytes_for_level_base=512m;level0_file_num_compaction_trigger=4;level0_slowdown_writes_trigger=10;level0_stop_writes_trigger=15;max_write_buffer_number=4;compression_per_level=kLZ4Compression;bottommost_compression=kZSTD;compression_opts=-14:1:0;block_based_table_factory= {cache_index_and_filter_blocks=1;filter_policy=bloomfilter:10:false;whole_key_filtering=1};level_compaction_dynamic_level_bytes=true;optimize_filters_for_hits=true;compaction_pri=kMinOverlappingRatio
Percona Server 5.7.19-17 Copied settings from facebook wiki page on server with 20GB RAM and 10GB swap: loose_rocksdb_max_open_files=-1 loose_rocksdb_max_background_jobs=8 loose_rocksdb_max_total_wal_size=4G loose_rocksdb_block_size=16384 loose_rocksdb_block_cache_size= 32G loose_rocksdb_table_cache_numshardbits=6 rate limiter loose_rocksdb_bytes_per_sync=4194304 loose_rocksdb_wal_bytes_per_sync=4194304 loose_rocksdb_rate_limiter_bytes_per_sec=104857600 #100MB/s. Increase if you're running on higher spec machines triggering compaction if there are many sequential deletes loose_rocksdb_compaction_sequential_deletes_count_sd=1 loose_rocksdb_compaction_sequential_deletes=199999 loose_rocksdb_compaction_sequential_deletes_window=200000 read free replication loose_rocksdb_rpl_lookup_rows=0 loose_rocksdb_default_cf_options=write_buffer_size=128m;target_file_size_base=32m;max_bytes_for_level_base=512m;level0_file_num_compaction_trigger=4;level0_slowdown_writes_trigger=10;level0_stop_writes_trigger=15;max_write_buffer_number=4;compression_per_level=kLZ4Compression;bottommost_compression=kZSTD;compression_opts=-14:1:0;block_based_table_factory= {cache_index_and_filter_blocks=1;filter_policy=bloomfilter:10:false;whole_key_filtering=1} ;level_compaction_dynamic_level_bytes=true;optimize_filters_for_hits=true;compaction_pri=kMinOverlappingRatio - rate limiter
alter table big_rocksdb_table ENGINE=ROCKSDB;
Database crashed due to out of memory:
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
18:18:42 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/
key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=2
max_threads=152
thread_count=1
connection_count=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 68315 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x7f01ac00d3c0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7f0220db0e80 thread_stack 0x40000
mysqld(my_print_stacktrace+0x2c)[0xe92eac]
mysqld(handle_fatal_signal+0x479)[0x79b0f9]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf890)[0x7f02b8c1a890]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37)[0x7f02b6b68067]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7f02b6b69448]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(ZN9gnu_cxx27_verbose_terminate_handlerEv+0x15d)[0x7f02b7463b3d]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5ebb6)[0x7f02b7461bb6]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5ec01)[0x7f02b7461c01]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5ee19)[0x7f02b7461e19]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x5f339)[0x7f02b7462339]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZNSs4_Rep9_S_createEmmRKSaIcE+0x59)[0x7f02b74c1e99]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZNSs4_Rep8_M_cloneERKSaIcEm+0x1b)[0x7f02b74c2b0b]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZNSs7reserveEm+0x30)[0x7f02b74c2bb0]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xbfe3a)[0x7f02b74c2e3a]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZNSs6appendEPKcm+0xe)[0x7f02b74c2f0e]
/usr/lib/mysql/plugin/ha_rocksdb.so(ZN7rocksdb18WriteBatchInternal3PutEPNS_10WriteBatchEjRKNS_5SliceES5+0x1a1)[0x7f01fdb01791]
/usr/lib/mysql/plugin/ha_rocksdb.so(ZN7rocksdb10WriteBatch3PutEPNS_18ColumnFamilyHandleERKNS_5SliceES5+0x36)[0x7f01fdb018e6]
/usr/lib/mysql/plugin/ha_rocksdb.so(ZN7rocksdb19WriteBatchWithIndex3PutEPNS_18ColumnFamilyHandleERKNS_5SliceES5+0x36)[0x7f01fdc7a326]
/usr/lib/mysql/plugin/ha_rocksdb.so(_ZN7myrocks10ha_rocksdb9update_pkERKNS_11Rdb_key_defERKNS0_15update_row_infoERKb+0xe6)[0x7f01fd99e896]
/usr/lib/mysql/plugin/ha_rocksdb.so(_ZN7myrocks10ha_rocksdb14update_indexesERKNS0_15update_row_infoERKb+0x41)[0x7f01fd99ea91]
/usr/lib/mysql/plugin/ha_rocksdb.so(_ZN7myrocks10ha_rocksdb16update_write_rowEPKhS2_b+0xda)[0x7f01fd9a208a]
/usr/lib/mysql/plugin/ha_rocksdb.so(_ZN7myrocks10ha_rocksdb9write_rowEPh+0x47)[0x7f01fd9a2117]
mysqld(_ZN7handler12ha_write_rowEPh+0x184)[0x804014]
mysqld[0x764e61]
mysqld(_Z17mysql_alter_tableP3THDPKcS2_P24st_ha_create_informationP10TABLE_LISTP10Alter_info+0x2edf)[0xcd918f]
mysqld(_ZN19Sql_cmd_alter_table7executeEP3THD+0x4ff)[0xdc657f]
mysqld(_Z21mysql_execute_commandP3THDb+0x1027)[0xc64447]
mysqld(_Z11mysql_parseP3THDP12Parser_state+0x625)[0xc69e15]
mysqld(_Z16dispatch_commandP3THDPK8COM_DATA19enum_server_command+0x942)[0xc6a7d2]
mysqld(_Z10do_commandP3THD+0x1b7)[0xc6c217]
mysqld(handle_connection+0x2a0)[0xd32960]
mysqld(pfs_spawn_thread+0x1b4)[0xeb2674]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8064)[0x7f02b8c13064]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f02b6c1b62d]
After restart ROCKSDB storage engine is unavailable:
2017-11-30T05:01:42.099938Z 0 [Warning] RocksDB: Schema mismatch - Table test.#sql-1_3 is registered in RocksDB but does not have a .frm file
2017-11-30T05:01:42.099956Z 0 [ERROR] RocksDB: Problems validating data dictionary against .frm files, exiting
2017-11-30T05:01:42.099960Z 0 [ERROR] RocksDB: Failed to initialize DDL manager.
2017-11-30T05:01:42.099982Z 0 [ERROR] Plugin 'ROCKSDB' init function returned error.
2017-11-30T05:01:42.099993Z 0 [ERROR] Plugin 'ROCKSDB' registration as a STORAGE ENGINE failed.