Uploaded image for project: 'Percona XtraDB Cluster'
  1. Percona XtraDB Cluster
  2. PXC-1997

LP #1706992: mysqld got signal 11


    • Bug
    • Status: Done
    • Low
    • Resolution: Cannot Reproduce
    • None
    • None
    • None


      **Reported in Launchpad by Brian Leishman last update 26-12-2017 07:45:36

      Something about our server is constantly getting crashes.

      Ubuntu 16.04 on AWS EC2
      Galera 3.20(r7e383f7)
      MySQL 5.7.18-15-57-log

      I have the log here


      14:15:12 UTC - mysqld got signal 11 ;
      This could be because you hit a bug. It is also possible that this binary
      or one of the libraries it was linked against is corrupt, improperly built,
      or misconfigured. This error can also be caused by malfunctioning hardware.
      Attempting to collect some information that could help diagnose the problem.
      As this is a crash and something is definitely wrong, the information
      collection process might fail.
      Please help us make Percona XtraDB Cluster better by reporting any
      bugs at https://bugs.launchpad.net/percona-xtradb-cluster

      It is possible that mysqld could use up to
      key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 207515 K bytes of memory
      Hope that's ok; if not, decrease some variables in the equation.

      Thread pointer: 0x7ff23c01daa0
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 7ff4f824ea70 thread_stack 0x40000

      Trying to get some variables.
      Some pointers may be invalid and cause the dump to abort.
      Query (7ff23cf141b8): is an invalid pointer
      Connection ID (thread ID): 8728
      Status: NOT_KILLED

      You may download the Percona XtraDB Cluster operations manual by visiting
      http://www.percona.com/software/percona-xtradb-cluster/. You may find information
      in the manual which will help you identify the cause of the crash.
      Log of wsrep recovery (--wsrep-recover):
      2017-07-27T14:15:28.000840Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
      2017-07-27T14:15:28.000905Z 0 [Warning] 'NO_ZERO_DATE', 'NO_ZERO_IN_DATE' and 'ERROR_FOR_DIVISION_BY_ZERO' sql modes should be used with strict mode. They will be merged with strict mode in a future release.
      2017-07-27T14:15:28.001925Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.18-15-57-log) starting as process 14618 ...
      2017-07-27T14:15:28.003973Z 0 [Warning] No argument was provided to -log-bin, and --log-bin-index was not used; so replication may break when this MySQL server acts as a master and has his hostname changed!! Please use '-log-bin=ip-172
      -31-24-62-bin' to avoid this problem.
      2017-07-27T14:15:28.005206Z 0 [Note] InnoDB: PUNCH HOLE support available
      2017-07-27T14:15:28.005219Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
      2017-07-27T14:15:28.005225Z 0 [Note] InnoDB: Uses event mutexes
      2017-07-27T14:15:28.005229Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
      2017-07-27T14:15:28.005233Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.8
      2017-07-27T14:15:28.005237Z 0 [Note] InnoDB: Using Linux native AIO
      2017-07-27T14:15:28.005617Z 0 [Note] InnoDB: Number of pools: 1
      2017-07-27T14:15:28.005733Z 0 [Note] InnoDB: Using CPU crc32 instructions
      2017-07-27T14:15:28.008009Z 0 [Note] InnoDB: Initializing buffer pool, total size = 9G, instances = 8, chunk size = 128M
      2017-07-27T14:15:28.257339Z 0 [Note] InnoDB: Completed initialization of buffer pool
      2017-07-27T14:15:28.318763Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
      2017-07-27T14:15:28.339004Z 0 [Note] InnoDB: Recovering partial pages from the parallel doublewrite buffer at /var/lib/mysql/xb_doublewrite
      2017-07-27T14:15:28.391505Z 0 [Note] InnoDB: Highest supported file format is Barracuda.
      2017-07-27T14:15:28.433258Z 0 [Note] InnoDB: Log scan progressed past the checkpoint lsn 19450410673
      2017-07-27T14:15:28.495023Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 19455653376
      2017-07-27T14:15:28.561670Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 19460896256
      2017-07-27T14:15:28.620811Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 19466139136
      2017-07-27T14:15:28.689284Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 19471382016
      2017-07-27T14:15:28.748074Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 19476624896
      2017-07-27T14:15:28.797094Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 19481867776
      2017-07-27T14:15:28.839447Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 19487110656
      2017-07-27T14:15:28.896844Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 19492353536
      2017-07-27T14:15:28.939504Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 19497596416
      2017-07-27T14:15:28.988672Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 19502839296
      2017-07-27T14:15:29.032153Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 19508082176
      2017-07-27T14:15:29.142124Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 19454998016
      2017-07-27T14:15:29.214843Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 19460240896
      2017-07-27T14:15:29.286710Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 19465483776
      2017-07-27T14:15:29.365828Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 19470726656
      2017-07-27T14:15:29.439394Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 19475969536
      2017-07-27T14:15:29.499052Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 19481212416
      2017-07-27T14:15:29.560015Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 19486455296
      2017-07-27T14:15:29.637035Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 19491698176
      2017-07-27T14:15:29.696519Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 19496941056
      2017-07-27T14:15:29.765000Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 19502183936
      2017-07-27T14:15:29.825511Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 19507426816
      2017-07-27T14:15:29.844417Z 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 19508869150
      2017-07-27T14:15:29.844449Z 0 [Note] InnoDB: Database was not shutdown normally!
      2017-07-27T14:15:29.844456Z 0 [Note] InnoDB: Starting crash recovery.
      2017-07-27T14:15:30.914759Z 0 [Note] InnoDB: Created parallel doublewrite buffer at /var/lib/mysql/xb_doublewrite, size 31457280 bytes
      2017-07-27T14:15:31.512861Z 0 [Note] InnoDB: Transaction 19972105 was in the XA prepared state.
      2017-07-27T14:15:31.525591Z 0 [Note] InnoDB: 1 transaction(s) which must be rolled back or cleaned up in total 0 row operations to undo
      2017-07-27T14:15:31.525603Z 0 [Note] InnoDB: Trx id counter is 19972608
      2017-07-27T14:15:31.525685Z 0 [Note] InnoDB: Starting an apply batch of log records to the database...
      InnoDB: Progress in percent: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72
      73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
      2017-07-27T14:15:33.920699Z 0 [Note] InnoDB: Apply batch completed
      2017-07-27T14:15:33.920747Z 0 [Note] InnoDB: Last MySQL binlog file position 0 37858379, file name ip-172-31-24-62-bin.000010
      2017-07-27T14:15:33.955536Z 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
      2017-07-27T14:15:33.955563Z 0 [Note] InnoDB: Creating shared tablespace for temporary tables
      2017-07-27T14:15:33.955537Z 0 [Note] InnoDB: Starting in background the rollback of uncommitted transactions
      2017-07-27T14:15:33.955584Z 0 [Note] InnoDB: Rollback of non-prepared transactions completed
      2017-07-27T14:15:33.955601Z 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
      2017-07-27T14:15:33.995268Z 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
      2017-07-27T14:15:33.995923Z 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active.
      2017-07-27T14:15:33.995933Z 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active.
      2017-07-27T14:15:33.996019Z 0 [Note] InnoDB: page_cleaner: 1000ms intended loop took 5678ms. The settings might not be optimal. (flushed=0, during the time.)
      2017-07-27T14:15:33.997102Z 0 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.7.18-15 started; log sequence number 19508869150
      2017-07-27T14:15:33.997117Z 0 [Warning] InnoDB: Skipping buffer pool dump/restore during wsrep recovery.
      2017-07-27T14:15:33.997369Z 0 [Note] Plugin 'FEDERATED' is disabled.
      2017-07-27T14:15:34.002326Z 0 [Note] Recovering after a crash using ip-172-31-24-62-bin
      2017-07-27T14:15:34.002345Z 0 [Note] WSREP: Binlog recovery, found wsrep position a148f201-7185-11e7-8dac-5232f5b724bd:9911775
      2017-07-27T14:15:34.186975Z 0 [Note] WSREP: Binlog recovery scan stopped at Xid event 9911775
      2017-07-27T14:15:34.187002Z 0 [Note] Starting crash recovery...
      2017-07-27T14:15:34.187030Z 0 [Note] InnoDB: Starting recovery for XA transactions...
      2017-07-27T14:15:34.187039Z 0 [Note] InnoDB: Transaction 19972105 in prepared state after recovery
      2017-07-27T14:15:34.187044Z 0 [Note] InnoDB: Transaction contains changes to 1 rows
      2017-07-27T14:15:34.187049Z 0 [Note] InnoDB: 1 transactions in prepared state after recovery
      2017-07-27T14:15:34.187053Z 0 [Note] Found 1 prepared transaction(s) in InnoDB
      2017-07-27T14:15:34.190019Z 0 [Note] Crash recovery finished.
      2017-07-27T14:15:34.190090Z 0 [Note] Crashed binlog file ./ip-172-31-24-62-bin.000010 size is 37858949, but recovered up to 37858379. Binlog trimmed to 37858379 bytes.
      2017-07-27T14:15:34.198806Z 0 [Note] WSREP: Recovered position: a148f201-7185-11e7-8dac-5232f5b724bd:9911775
      2017-07-27T14:15:34.198827Z 0 [Note] Binlog end
      2017-07-27T14:15:34.201281Z 0 [Note] Shutting down plugin 'ngram'
      2017-07-27T14:15:34.201291Z 0 [Note] Shutting down plugin 'ARCHIVE'
      2017-07-27T14:15:34.201294Z 0 [Note] Shutting down plugin 'partition'
      2017-07-27T14:15:34.201297Z 0 [Note] Shutting down plugin 'BLACKHOLE'
      2017-07-27T14:15:34.201301Z 0 [Note] Shutting down plugin 'PERFORMANCE_SCHEMA'
      2017-07-27T14:15:34.201347Z 0 [Note] Shutting down plugin 'MyISAM'
      2017-07-27T14:15:34.201359Z 0 [Note] Shutting down plugin 'CSV'
      2017-07-27T14:15:34.201364Z 0 [Note] Shutting down plugin 'MRG_MYISAM'
      2017-07-27T14:15:34.201367Z 0 [Note] Shutting down plugin 'MEMORY'
      2017-07-27T14:15:34.201371Z 0 [Note] Shutting down plugin 'INNODB_SYS_VIRTUAL'
      2017-07-27T14:15:34.201374Z 0 [Note] Shutting down plugin 'INNODB_CHANGED_PAGES'
      2017-07-27T14:15:34.201377Z 0 [Note] Shutting down plugin 'INNODB_SYS_DATAFILES'
      2017-07-27T14:15:34.201379Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLESPACES'
      2017-07-27T14:15:34.201382Z 0 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN_COLS'
      2017-07-27T14:15:34.201384Z 0 [Note] Shutting down plugin 'INNODB_SYS_FOREIGN'
      2017-07-27T14:15:34.201387Z 0 [Note] Shutting down plugin 'INNODB_SYS_FIELDS'
      2017-07-27T14:15:34.201389Z 0 [Note] Shutting down plugin 'INNODB_SYS_COLUMNS'
      2017-07-27T14:15:34.201392Z 0 [Note] Shutting down plugin 'INNODB_SYS_INDEXES'
      2017-07-27T14:15:34.201394Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLESTATS'
      2017-07-27T14:15:34.201397Z 0 [Note] Shutting down plugin 'INNODB_SYS_TABLES'
      2017-07-27T14:15:34.201399Z 0 [Note] Shutting down plugin 'INNODB_FT_INDEX_TABLE'
      2017-07-27T14:15:34.201402Z 0 [Note] Shutting down plugin 'INNODB_FT_INDEX_CACHE'
      2017-07-27T14:15:34.201404Z 0 [Note] Shutting down plugin 'INNODB_FT_CONFIG'
      2017-07-27T14:15:34.201407Z 0 [Note] Shutting down plugin 'INNODB_FT_BEING_DELETED'
      2017-07-27T14:15:34.201409Z 0 [Note] Shutting down plugin 'INNODB_FT_DELETED'
      2017-07-27T14:15:34.201412Z 0 [Note] Shutting down plugin 'INNODB_FT_DEFAULT_STOPWORD'
      2017-07-27T14:15:34.201414Z 0 [Note] Shutting down plugin 'INNODB_METRICS'
      2017-07-27T14:15:34.201417Z 0 [Note] Shutting down plugin 'INNODB_TEMP_TABLE_INFO'
      2017-07-27T14:15:34.201419Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_POOL_STATS'
      2017-07-27T14:15:34.201422Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE_LRU'
      2017-07-27T14:15:34.201424Z 0 [Note] Shutting down plugin 'INNODB_BUFFER_PAGE'
      2017-07-27T14:15:34.201427Z 0 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX_RESET'
      2017-07-27T14:15:34.201429Z 0 [Note] Shutting down plugin 'INNODB_CMP_PER_INDEX'
      2017-07-27T14:15:34.201432Z 0 [Note] Shutting down plugin 'INNODB_CMPMEM_RESET'
      2017-07-27T14:15:34.201434Z 0 [Note] Shutting down plugin 'INNODB_CMPMEM'
      2017-07-27T14:15:34.201437Z 0 [Note] Shutting down plugin 'INNODB_CMP_RESET'
      2017-07-27T14:15:34.201457Z 0 [Note] Shutting down plugin 'INNODB_CMP'
      2017-07-27T14:15:34.201460Z 0 [Note] Shutting down plugin 'INNODB_LOCK_WAITS'
      2017-07-27T14:15:34.201463Z 0 [Note] Shutting down plugin 'INNODB_LOCKS'
      2017-07-27T14:15:34.201465Z 0 [Note] Shutting down plugin 'INNODB_TRX'
      2017-07-27T14:15:34.201468Z 0 [Note] Shutting down plugin 'XTRADB_ZIP_DICT_COLS'
      2017-07-27T14:15:34.201471Z 0 [Note] Shutting down plugin 'XTRADB_ZIP_DICT'
      2017-07-27T14:15:34.201473Z 0 [Note] Shutting down plugin 'XTRADB_RSEG'
      2017-07-27T14:15:34.201476Z 0 [Note] Shutting down plugin 'XTRADB_INTERNAL_HASH_TABLES'
      2017-07-27T14:15:34.201478Z 0 [Note] Shutting down plugin 'XTRADB_READ_VIEW'
      2017-07-27T14:15:34.201481Z 0 [Note] Shutting down plugin 'InnoDB'
      2017-07-27T14:15:34.201518Z 0 [Note] InnoDB: FTS optimize thread exiting.
      2017-07-27T14:15:34.201580Z 0 [Note] InnoDB: Starting shutdown...
      2017-07-27T14:15:35.302997Z 0 [Note] InnoDB: Waiting for page_cleaner to finish flushing of buffer pool
      2017-07-27T14:15:37.922712Z 0 [Note] InnoDB: Shutdown completed; log sequence number 19508876239
      2017-07-27T14:15:37.922801Z 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
      2017-07-27T14:15:37.922812Z 0 [Note] Shutting down plugin 'sha256_password'
      2017-07-27T14:15:37.922826Z 0 [Note] Shutting down plugin 'mysql_native_password'
      2017-07-27T14:15:37.922830Z 0 [Note] Shutting down plugin 'wsrep'
      2017-07-27T14:15:37.922974Z 0 [Note] Shutting down plugin 'binlog'
      2017-07-27T14:15:37.923161Z 0 [Note] /usr/sbin/mysqld: Shutdown complete


      And this is a fresh install, so all the default Percona config files are still there, except I removed the wsrep.cnf file in /etc/mysql/percona-xtradb-cluster.conf.d, and I've added the following to /etc/mysql/my.cnf


      default-character-set = utf8mb4

      default-character-set = utf8mb4



      max-allowed-packet = 16M
      max-connect-errors = 1000000

      tmp-table-size = 32M
      max-heap-table-size = 32M
      query-cache-type = 0
      query-cache-size = 0
      max-connections = 500
      thread-cache-size = 50
      open-files-limit = 65535
      table-definition-cache = 1024
      table-open-cache = 2048

      innodb-flush-method = O_DIRECT
      innodb-log-files-in-group = 2
      innodb-log-file-size = 256M
      innodb-flush-log-at-trx-commit = 1
      innodb-file-per-table = 1
      innodb-buffer-pool-size = 9G









      Most of the settings were either from the guide for setting up the cluster on Ubuntu, and from the Percona MySQL config wizard




            Unassigned Unassigned
            lpjirasync lpjirasync (Inactive)
            0 Vote for this issue
            2 Start watching this issue



              Smart Checklist