Uploaded image for project: 'Percona XtraDB Cluster'
  1. Percona XtraDB Cluster
  2. PXC-3367

Warning "The SSL library function CRYPTO_set_mem_functions failed" and openssl_malloc P_S memory event missing when using PXC8

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: On Hold
    • Priority: Low
    • Resolution: Unresolved
    • Affects Version/s: 8.0.19-10
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      When using PXC8, the following warning is printed in the logs on every startup.

      2020-07-20T14:31:50.844521Z 0 [Warning] [MY-013245] [Server] The SSL library function CRYPTO_set_mem_functions failed. This is typically caused by the SSL library already being used. As a result the SSL memory allocation will not be instrumented.
      

      As a consequence, P_S memory event memory/mysqld_openssl/openssl_malloc is missing.

      To reproduce, simply start PXC8, a warning will be present in MySQL's error log, and openssl_malloc is missing.

      mysql> SELECT * FROM sys.memory_global_by_current_bytes WHERE LOWER(event_name) LIKE '%ssl%';
      +-------------------+---------------+---------------+-------------------+------------+------------+----------------+
      | event_name        | current_count | current_alloc | current_avg_alloc | high_count | high_alloc | high_avg_alloc |
      +-------------------+---------------+---------------+-------------------+------------+------------+----------------+
      | memory/vio/ssl_fd |             2 |   16 bytes    |    8 bytes        |          2 |   16 bytes |    8 bytes     |
      +-------------------+---------------+---------------+-------------------+------------+------------+----------------+
      1 row in set (0.01 sec)
      

      If wsrep_provider is commented out, the warning is gone, and instrumentation is present.

      mysql> select * from sys.memory_global_by_current_bytes where lower(event_name) like '%ssl%';
      +--------------------------------------+---------------+---------------+-------------------+------------+------------+----------------+
      | event_name                           | current_count | current_alloc | current_avg_alloc | high_count | high_alloc | high_avg_alloc |
      +--------------------------------------+---------------+---------------+-------------------+------------+------------+----------------+
      | memory/mysqld_openssl/openssl_malloc |          4179 | 152.67 KiB    |   37 bytes        |       4196 | 156.40 KiB |   38 bytes     |
      | memory/vio/ssl_fd                    |             2 |   16 bytes    |    8 bytes        |          2 |   16 bytes |    8 bytes     |
      +--------------------------------------+---------------+---------------+-------------------+------------+------------+----------------+
      2 rows in set (0.01 sec)
      

      The reason for this behavior is that mysqld will try to call CRYPTO_set_mem_functions() and pass custom allocation wrappers to it, see here https://github.com/percona/percona-xtradb-cluster/blob/Percona-XtraDB-Cluster-8.0.19-10.1/sql/mysqld.cc#L5623 However, CRYPTO_set_mem_functions() apparently returns 0. The behavior is same in OpenSSL 1.0.2k and 1.1.1g, so this definitely happens because allow_customize is 0 here https://github.com/openssl/openssl/blob/OpenSSL_1_1_1g/crypto/mem.c#L66 That seems to only be possible when CRYPTO_malloc() has already been called, as described here https://github.com/openssl/openssl/blob/OpenSSL_1_1_1g/crypto/mem.c#L210

      This is easy to verify with ltrace.

      PXC8 with wsrep_provider not set or a regular PS
      # ltrace -C -f -e CRYPTO_malloc -e CRYPTO_set_mem_functions -r /usr/sbin/mysqld --user=root --skip-stack
      -trace
      [pid 31329]   0.000000 mysqld->CRYPTO_set_mem_functions(0xf6e340, 0xf6e470, 0xf6e370, 0x4085bc0) = 1
      [pid 31329]   0.027346 libssl.so.1.1->CRYPTO_malloc(24, 0x7f684198faa8, 469, 0x40bd690) = 0x408ebd0
      
      PXC8 with wsrep_provider set
      # ltrace -C -f -e CRYPTO_malloc -e CRYPTO_set_mem_functions -r /usr/sbin/mysqld --user=root --skip-stack-trace
      [pid 31376]   0.000000 libssl.so.1.1->CRYPTO_malloc(24, 0x7ff50a42eaa8, 469, 0x4ddb8b0) = 0x4dc1290
      [pid 31376]   0.020032 libssl.so.1.1->CRYPTO_malloc(143, 0x7ff50a42eaa8, 1442, 56) = 0x4df2070
      [pid 31376]   0.001098 libssl.so.1.1->CRYPTO_malloc(5952, 0x7ff50a42eaa8, 1508, 8) = 0x4dfd730
      [pid 31376]   0.000855 libssl.so.1.1->CRYPTO_malloc(2104, 0x7ff50a42eaa8, 1620, 0x4dfecd0) = 0x4e2ea60
      [pid 31376]   0.002946 libssl.so.1.1->CRYPTO_malloc(5952, 0x7ff50a42eaa8, 1508, 8) = 0x4e2ea60
      [pid 31376]   0.001503 libssl.so.1.1->CRYPTO_malloc(2104, 0x7ff50a42eaa8, 1620, 0x4e30000) = 0x4dfe270
      [pid 31376]   0.001995 libssl.so.1.1->CRYPTO_malloc(36, 0x7ff50a436118, 2083, 0) = 0x4dfd820
      [pid 31376]   0.058755 libssl.so.1.1->CRYPTO_malloc(143, 0x7ff50a42eaa8, 1442, 56) = 0x4ddc950
      [pid 31376]   0.001298 libssl.so.1.1->CRYPTO_malloc(5952, 0x7ff50a42eaa8, 1508, 8) = 0x4e374b0
      [pid 31376]   0.001268 libssl.so.1.1->CRYPTO_malloc(2104, 0x7ff50a42eaa8, 1620, 0x4e38a50) = 0x4e35a20
      [pid 31376]   0.002175 libssl.so.1.1->CRYPTO_malloc(5952, 0x7ff50a42eaa8, 1508, 8) = 0x4e374b0
      [pid 31376]   0.001712 libssl.so.1.1->CRYPTO_malloc(2104, 0x7ff50a42eaa8, 1620, 0x4e38a50) = 0x4e35cb0
      [pid 31376]   0.001944 libssl.so.1.1->CRYPTO_malloc(36, 0x7ff50a436118, 2083, 0) = 0x4e35cb0
      [pid 31376]   0.070389 mysqld->CRYPTO_set_mem_functions(0xf6e340, 0xf6e470, 0xf6e370, 0x4e196e0) = 0
      

      Breaking at CRYPTO_malloc shows that the first call (at least) is done as part of wsrep_init(). See gdb.txt attached.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              Unassigned
              Reporter:
              sergey.kuzmichev Sergey Kuzmichev
              Votes:
              2 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:

                  Smart Checklist