Uploaded image for project: 'Percona Server'
  1. Percona Server
  2. PS-5089

RocksDB index creation terminates server due to lack of memory

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: On Hold
    • Priority: Critical
    • Resolution: Reopened
    • Affects Version/s: 5.7, 8.0
    • Fix Version/s: None
    • Component/s: MyRocks
    • Labels:

      Description

      RocksDB index creation terminates server due to lack of memory.

      Testcase

      # DB initialization
      /data/perf/Percona-Server-5.7.23-23-Linux.x86_64.ssl101/bin/mysqld --no-defaults --initialize-insecure --basedir=/data/perf/Percona-Server-5.7.23-23-Linux.x86_64.ssl101 --datadir=/data/perf/Percona-Server-5.7.23-23-Linux.x86_64.ssl101/data
      
      # DB startup.
      
      /data/perf/Percona-Server-5.7.23-23-Linux.x86_64.ssl101/bin/mysqld --defaults-file=/home/ramesh/toku-qa/bin/my_rocks.cnf --user=ramesh --basedir=/data/perf/Percona-Server-5.7.23-23-Linux.x86_64.ssl101 --datadir=/data/perf/Percona-Server-5.7.23-23-Linux.x86_64.ssl101/data --rocksdb-block-cache-size=10G --plugin-load-add=rocksdb=ha_rocksdb.so --init-file=/home/ramesh/toku-qa/bin/MyRocks.sql --default-storage-engine=ROCKSDB --port=30000 --pid-file=/data/perf/Percona-Server-5.7.23-23-Linux.x86_64.ssl101/data/pid.pid --core-file --socket=/tmp/ps-perf-mysql.sock --log-error=/data/perf/Percona-Server-5.7.23-23-Linux.x86_64.ssl101/data/error.log.out --log_error_verbosity=3  &
      
      Attached my.cnf
      
      # Create sysbench user
      /data/perf/Percona-Server-5.7.23-23-Linux.x86_64.ssl101/bin/mysql -uroot -S/tmp/ps-perf-mysql.sock -e"CREATE USER IF NOT EXISTS sysbench_user@'%' identified with mysql_native_password by 'test';GRANT ALL ON *.* TO sysbench_user@'%';"
      
      # create test database
      /data/perf/Percona-Server-5.7.23-23-Linux.x86_64.ssl101/bin/mysql -uroot -S/tmp/ps-perf-mysql.sock -e'DROP DATABASE IF EXISTS test; CREATE DATABASE test;'
      
      # Sysbench prepare
      
      sysbench /usr/share/sysbench/oltp_insert.lua --mysql-storage-engine=rocksdb --rand-type=uniform --threads=16  --tables=16  --table-size=100000 --mysql-db=test --mysql-user=sysbench_user --mysql-password=test --db-driver=mysql --mysql-socket=/tmp/ps-perf-mysql.sock --create_secondary=off prepare
      
      
      start vmstat in different session
      vmstat -S M 2
      
      
      # Run create index sql
      
      for i in `seq 1 16`;do  
        /data/perf/Percona-Server-5.7.23-23-Linux.x86_64.ssl101/bin/mysql -uroot -S/tmp/ps-perf-mysql.sock test -e"CREATE INDEX k_${i} ON sbtest${i}(k);" & 
      done
      
      

      vmstat output

      [ramesh@dell-r420 ~]$ vmstat -S M 2
      procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
       r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
       0  0    127  46344    414    571    0    0     1     9    0    0  0  0 100  0  0
       0  0    127  46344    414    571    0    0     0     7  120  192  0  0 100  0  0
       0  0    127  46344    414    571    0    0     0     0  162  186  0  0 100  0  0
       0  0    127  46344    414    571    0    0     0    16  151  233  0  0 100  0  0
       0  0    127  46344    414    571    0    0     0     0  103  174  0  0 100  0  0
       0  0    127  46344    414    571    0    0     0     0  138  352  0  0 100  0  0
       0  0    127  46344    414    571    0    0     0     1  135  195  0  0 100  0  0
       0  0    127  46344    414    571    0    0     0     0  113  171  0  0 100  0  0
       0  0    127  46344    414    571    0    0     0     0  143  191  0  0 100  0  0
       0  0    127  46344    414    571    0    0     0     0  111  171  0  0 100  0  0
      16  0    127  38215    414    571    0    0    20   368 7778 1102  5 18 77  0  0
      16  0    127  21581    414    571    0    0     0     0 16309  173 10 40 50  0  0
      16  0    127   5975    414    571    0    0     0     0 16577  167  9 41 50  0  0
      17  0    127    257    165    231    0    0     0   152 2082408  336  4 48 48  0  0
      19  0    257    257      0     26    0   66     2 67952 2031510 155096  0 37 54  9  0
      14  2    457    257      0     28    1  100  1056 102207 625191 432865  0 22 68 11  0
       4  2    675    257      0     24    0  109   348 111700 782466 520294  0 28 65  6  0
      19  1    915    258      0     28    0  120    16 123249 896619 527909  0 31 65  3  0
       9  2   1121    257      0     26    1  103   896 105412 738324 465577  0 26 67  7  0
      18  1   1281    257      1     31    2   80  3738 82290 573603 452298  0 25 65  9  0
       0  2   1441    257      1     31    1   80  2160 81635 542354 342004  0 21 71  8  0
      13  4   1649    257      1     26    2  104  2782 106820 763591 494964  0 31 59  9  0
       0  3   1859    257      1     26    1  105   864 107700 742735 489906  0 28 65  7  0
       0  1   2053    257      1     27    0   97   296 99561 699229 454911  0 24 72  4  0
       5  1   2298    257      1     24    0  122    16 125453 890199 501556  0 29 67  3  0
       9  2   2523    257      1     24    0  112    34 115085 857038 505328  0 30 67  3  0
      14  2   2749    257      1     26    0  113     0 115472 840411 516852  0 29 67  4  0
      procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
       r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
       9  2   2971    257      1     25    0  111    50 113972 774706 493694  0 28 69  3  0
      16  2   3171    257      1     28    0  100  1058 102530 743303 471558  0 26 69  4  0
       0  1   3368    257      1     28    1   98  1004 100765 717766 474552  0 26 68  5  0
       0  2   3578    257      2     26    1  105   870 107723 797522 487601  0 30 67  4  0
       0  1   3793    257      2     30    0  107    16 109757 831212 517438  0 30 67  3  0
      10  2   4016    257      2     29    0  112   732 114623 827276 487724  0 30 66  4  0
      18  2   4209    257      2     28    0   97   660 98851 723049 483460  0 28 67  5  0
       0  1   4419    257      3     25    0  105   854 107533 772962 486360  0 28 67  5  0
       0  1   4659    257      3     29    0  120    12 123021 889924 514883  0 30 67  3  0
       5  1   4890    257      3     27    0  115    14 118132 874881 524811  0 30 67  3  0
       0  1   5118    257      3     24    0  114   128 116823 850759 509817  0 30 65  4  0
      10  1   5339    257      3     26    0  111    20 113358 831909 502630  0 30 67  3  0
      19  1   5566    257      3     27    0  114     0 116328 855416 521416  0 30 67  3  0
       0  2   5783    257      3     24    0  108     0 110864 832390 528972  0 30 66  4  0
      11  2   6007    258      3     23    0  112    16 114552 769470 498063  0 26 70  4  0
       0  2   6232    257      3     24    0  113    16 115327 842841 515130  0 29 67  4  0
      17  1   6449    257      3     25    0  109     0 111175 812432 505114  0 29 68  3  0
       0  1   6663    257      3     25    0  107    16 109623 812501 506920  0 29 67  4  0
      17  0   6880    258      3     26    0  108    16 110992 831847 507945  0 30 67  3  0
       0  1   7133    257      3     23    0  126     0 129515 959680 535982  0 33 65  2  0
      17  1   7359    257      3     26    0  113    16 116104 884528 560949  0 31 65  3  0
      17  1   7576    258      3     25    0  108     0 111108 796629 510182  0 28 68  3  0
       1  2   7815    257      3     24    0  119    32 122211 915889 538286  0 33 64  3  0
       0  2   7918    258      3     26    0   54    86 54926 515842 460513  0 21 74  5  0
      12  1   8004    257      3     25    0   47   112 47809 329581 198230  0 11 85  4  0
       0  1   8086    258      3     25    1   42   772 43379 266044 240021  0 12 81  7  0
       0  1   8148    257      3     23    0   31   564 32180 167219 109603  0  5 90  5  0
      procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
       r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
       3  2   6213   9522      1      7    1   23  3310 24002 249799 110368  0 13 81  6  0
       2  1   1813  38439      1      7    0    8   596  8152 4672 3099  0  3 93  3  0
       0  0    141  47760      1     12    5    4  8016  3640 2976 3657  0  2 92  7  0
       0  0    141  47761      1     11    0    0     2     0  101  149  0  0 100  0  0
       0  0    141  47762      1     11    0    0     4     0   85  121  0  0 100  0  0
       0  0    141  47761      1     14    0    0  1272    23  163  312  0  0 100  0  0
       0  0    141  47759      1     17    0    0  2084     0  249  415  0  0 99  1  0
       0  0    141  47758      1     17    0    0    32     0  115  152  0  0 100  0  0
       0  0    141  47755      1     19    0    0  1608     0  281  393  0  0 99  1  0
       0  0    141  47755      1     20    0    0     2     0  109  167  0  0 100  0  0
       0  0    141  47755      1     20    0    0     2     1   80  115  0  0 100  0  0
       0  0    141  47755      1     20    0    0     4     9  107  178  0  0 100  0  0
       0  0    141  47755      1     20    0    0     2     0  105  127  0  0 100  0  0
       0  0    141  47755      1     20    0    0    36     0  103  137  0  0 100  0  0
      ^C
      [ramesh@dell-r420 ~]$
       

        Smart Checklist

          Attachments

            Issue Links

              Activity

                People

                • Assignee:
                  george.lorch George Lorch
                  Reporter:
                  ramesh.sivaraman Ramesh Sivaraman
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  4 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Time Tracking

                    Estimated:
                    Original Estimate - Not Specified
                    Not Specified
                    Remaining:
                    Remaining Estimate - 0 minutes
                    0m
                    Logged:
                    Time Spent - 15 minutes
                    15m