Details

    • Bug
    • Status: Done
    • Medium
    • Resolution: Fixed
    • 2.4.x, 8.0.x
    • 8.0.27-19 (Q4 2021), 2.4.25
    • None
    • None
    • Yes
    • Yes

    Description

      In short:
      PXB still uses Vault API v1, while PS already has support for V2

      I'm using Percona Mysql Server 8.0.25 and Xtrabackup 8.0.25 (O.S: Debian 10). The database is in data-at-rest mode, with configuration bellow:

       

      early-plugin-load="keyring_vault=keyring_vault.so"
      loose-keyring_vault_config="/opt/vitess/examples/local/vault_configs/voice_abc/-80/keyring_vault.conf"
      binlog_encryption=ON
      binlog_rotate_encryption_master_key_at_startup=OFF
      default_table_encryption=ON
      encrypt_tmp_files=ON
      innodb_encrypt_online_alter_logs=ON
      table_encryption_privilege_check=ON
      innodb_redo_log_encrypt=ON
      innodb_undo_log_encrypt=ON
      innodb_temp_tablespace_encrypt=ON
      

       

       

       

      $ cat /opt/vitess/examples/local/vault_configs/voice_abc/-80/keyring_vault.conf
      vault_url = https://10.128.0.17:8200
      secret_mount_point = secrets/google/voice_abc/-80
      secret_mount_point_version = 1
      token = xxxxxxxx
      vault_ca = /opt/vitess/examples/local/vault_configs/vault_ca.crt
      

       

      Encryption works fine, but when a try to make a backup using xtrabackup I'm getting error bellow:

       

       

      suporte@mysql-02:/opt/vitess/examples/local$ xtrabackup --backup --datadir=/vt/vtdataroot/vt_0000000101/data/ --target-dir=/tmp/BKP/ -u xtrabackup_vitess -p -S /vt/vtdataroot/vt_0000000101/mysql.sock --keyring-vault-config=/opt/vitess/examples/local/vault_configs/voice_abc/-80/keyring_vault.conf
      xtrabackup: recognized server arguments: --datadir=/var/lib/mysql --datadir=/vt/vtdataroot/vt_0000000101/data/
      xtrabackup: recognized client arguments: --backup=1 --target-dir=/tmp/BKP/ --user=xtrabackup_vitess --password --socket=/vt/vtdataroot/vt_0000000101/mysql.sock
      Enter password:
      xtrabackup version 8.0.25-17 based on MySQL server 8.0.25 Linux (x86_64) (revision id: d27028b)
      210909 16:04:34  version_check Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_group=xtrabackup;mysql_socket=/vt/vtdataroot/vt_0000000101/mysql.sock' as 'xtrabackup_vitess'  (using password: YES).
      210909 16:04:34  version_check Connected to MySQL server
      210909 16:04:34  version_check Executing a version check against the server...
      210909 16:04:34  version_check Done.
      210909 16:04:34 Connecting to MySQL server host: localhost, user: xtrabackup_vitess, password: set, port: not set, socket: /vt/vtdataroot/vt_0000000101/mysql.sock
      Using server version 8.0.25-15
      210909 16:04:34 Executing LOCK TABLES FOR BACKUP...
      xtrabackup: uses posix_fadvise().
      xtrabackup: cd to /vt/vtdataroot/vt_0000000101/data/
      xtrabackup: open files limit requested 0, set to 1024
      xtrabackup: using the following InnoDB configuration:
      xtrabackup:   innodb_data_home_dir = /vt/vtdataroot/vt_0000000101/innodb/data
      xtrabackup:   innodb_data_file_path = ibdata1:12M:autoextend
      xtrabackup:   innodb_log_group_home_dir = /vt/vtdataroot/vt_0000000101/innodb/logs
      xtrabackup:   innodb_log_files_in_group = 2
      xtrabackup:   innodb_log_file_size = 50331648
      Number of pools: 1
      210909 16:04:34 Added plugin 'keyring_vault.so' to load list.
      2021-09-09T16:04:34.076080Z 0 [System] [MY-011197] [InnoDB] Plugin keyring_vault reported: 'keyring_vault initialization failure. Please check that the keyring_vault_config_file points to readable keyring_vault configuration file. Please also make sure Vault is running and accessible. The keyring_vault will stay unusable until correct configuration file gets provided.'
      xtrabackup: inititialize_service_handles suceeded
      Encryption can't find master key, please check the keyring is loaded.
      Cannot read the encryption information in log file header, please check if keyring is loaded.
      210909 16:04:34 Connecting to MySQL server host: localhost, user: xtrabackup_vitess, password: set, port: not set, socket: /vt/vtdataroot/vt_0000000101/mysql.sock
      xtrabackup: Redo Log Archiving is not set up.
      xtrabackup: Can't create/write to file '/tmp/BKP/xtrabackup_logfile' (OS errno 17 - File exists)
      xtrabackup: error: failed to open the target stream for 'xtrabackup_logfile'.
      suporte@mysql-02:/opt/vitess/examples/local$
      
      

       
      Vault is working properly and is accessible, and configuration file is readable. I try too specify the pluguin directore of percona xtrabackup, but the same error occurs:

       

      suporte@mysql-02:/opt/vitess/examples/local$ sudo dpkg -L percona-xtrabackup-80 | grep keyring_vault.so
      /usr/lib/xtrabackup/plugin/keyring_vault.so
      
      suporte@mysql-02:/opt/vitess/examples/local$ locate keyring_vault.so
      /usr/lib/mysql/plugin/keyring_vault.so
      /usr/lib/mysql/plugin/debug/keyring_vault.so
      /usr/lib/xtrabackup/plugin/keyring_vault.so
      suporte@mysql-02:/opt/vitess/examples/local$ ls -l /tmp/BKP/
      total 0
      
      
      
      suporte@mysql-02:/opt/vitess/examples/local$ xtrabackup --backup --datadir=/vt/vtdataroot/vt_0000000101/data/ --target-dir=/tmp/BKP/ -u xtrabackup_vitess -p -S /vt/vtdataroot/vt_0000000101/mysql.sock --keyring-vault-config=/opt/vitess/examples/local/vault_configs/voice_abc/-80/keyring_vault.conf --xtrabackup-plugin-dir=/usr/lib/xtrabackup/plugin/
      xtrabackup: recognized server arguments: --datadir=/var/lib/mysql --datadir=/vt/vtdataroot/vt_0000000101/data/
      xtrabackup: recognized client arguments: --backup=1 --target-dir=/tmp/BKP/ --user=xtrabackup_vitess --password --socket=/vt/vtdataroot/vt_0000000101/mysql.sock --xtrabackup-plugin-dir=/usr/lib/xtrabackup/plugin/
      Enter password:
      xtrabackup version 8.0.25-17 based on MySQL server 8.0.25 Linux (x86_64) (revision id: d27028b)
      210909 17:13:27  version_check Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_group=xtrabackup;mysql_socket=/vt/vtdataroot/vt_0000000101/mysql.sock' as 'xtrabackup_vitess'  (using password: YES).
      210909 17:13:27  version_check Connected to MySQL server
      210909 17:13:27  version_check Executing a version check against the server...
      210909 17:13:27  version_check Done.
      210909 17:13:27 Connecting to MySQL server host: localhost, user: xtrabackup_vitess, password: set, port: not set, socket: /vt/vtdataroot/vt_0000000101/mysql.sock
      Using server version 8.0.25-15
      210909 17:13:27 Executing LOCK TABLES FOR BACKUP...
      xtrabackup: uses posix_fadvise().
      xtrabackup: cd to /vt/vtdataroot/vt_0000000101/data/
      xtrabackup: open files limit requested 0, set to 1024
      xtrabackup: using the following InnoDB configuration:
      xtrabackup:   innodb_data_home_dir = /vt/vtdataroot/vt_0000000101/innodb/data
      xtrabackup:   innodb_data_file_path = ibdata1:12M:autoextend
      xtrabackup:   innodb_log_group_home_dir = /vt/vtdataroot/vt_0000000101/innodb/logs
      xtrabackup:   innodb_log_files_in_group = 2
      xtrabackup:   innodb_log_file_size = 50331648
      Number of pools: 1
      210909 17:13:27 Added plugin 'keyring_vault.so' to load list.
      2021-09-09T17:13:27.695838Z 0 [System] [MY-011197] [InnoDB] Plugin keyring_vault reported: 'keyring_vault initialization failure. Please check that the keyring_vault_config_file points to readable keyring_vault configuration file. Please also make sure Vault is running and accessible. The keyring_vault will stay unusable until correct configuration file gets provided.'
      xtrabackup: inititialize_service_handles suceeded
      Encryption can't find master key, please check the keyring is loaded.
      Cannot read the encryption information in log file header, please check if keyring is loaded.
      210909 17:13:27 Connecting to MySQL server host: localhost, user: xtrabackup_vitess, password: set, port: not set, socket: /vt/vtdataroot/vt_0000000101/mysql.sock
      xtrabackup: Redo Log Archiving is not set up.
      Encryption algorithm support missing: N
      InnoDB: Assertion failure: redo_log.cc:161:err == DB_SUCCESS
      InnoDB: thread 139901056161792InnoDB: We intentionally generate a memory trap.
      InnoDB: Submit a detailed bug report to https://jira.percona.com/projects/PXB.
      InnoDB: If you get repeated assertion failures or crashes, even
      InnoDB: immediately after the mysqld startup, there may be
      InnoDB: corruption in the InnoDB tablespace. Please refer to
      InnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html
      InnoDB: about forcing recovery.
      17:13:27 UTC - mysqld got signal 6 ;
      Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.
      Thread pointer: 0x0
      Attempting backtrace. You can use the following information to find out
      where mysqld died. If you see no messages after this, something went
      terribly wrong...
      stack_bottom = 0 thread_stack 0x46000
      xtrabackup(my_print_stacktrace(unsigned char const*, unsigned long)+0x2e) [0x558b0e9d335e]
      xtrabackup(handle_fatal_signal+0x31b) [0x558b0d8bf9bb]
      /lib/x86_64-linux-gnu/libpthread.so.0(+0x12730) [0x7f3d4163a730]
      /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x10b) [0x7f3d40c557bb]
      /lib/x86_64-linux-gnu/libc.so.6(abort+0x121) [0x7f3d40c40535]
      xtrabackup(+0xe2285a) [0x558b0d32485a]
      xtrabackup(+0xefab88) [0x558b0d3fcb88]
      xtrabackup(Redo_Log_Reader::read_logfile(bool, bool*)+0x7f) [0x558b0d3fce7f]
      xtrabackup(Redo_Log_Data_Manager::copy_once(bool, bool*)+0x3f) [0x558b0d3fdb6f]
      xtrabackup(Redo_Log_Data_Manager::start()+0x9d) [0x558b0d3fdd9d]
      xtrabackup(xtrabackup_backup_func()+0x4e0) [0x558b0d3be9e0]
      xtrabackup(main+0x1670) [0x558b0d37cc30]
      /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb) [0x7f3d40c4209b]
      xtrabackup(_start+0x2a) [0x558b0d3ac53a]
      
      Please report a bug at https://jira.percona.com/projects/PXB
      suporte@mysql-02:/opt/vitess/examples/local$
      

       
       This is the link of an strace:
       
      https://fortics-my.sharepoint.com/:u:/g/personal/benedito_fortics_com_br/Ec0e_f8983JIqg3tFIKkXUMBhfUOAUzkFD5srqneRok3Mw?e=bxwjYz
       
      After that, I've been trying to follow the steps bellow, suggested on percona community:
       

      • Copy /opt/vitess/examples/local/vault_configs/voice_abc/-80/keyring_vault.conf to a different file
      • Edit the nes file and remove option secret_mount_point_version
      • Take the backup again passing the new vault.cnf file.
         
        Same error occurs:
         
        suporte@mysql-01:/opt/vitess/examples/local$ cp /opt/vitess/examples/local/vault_configs/voice_abc/-80/keyring_vault.conf /opt/vitess/examples/local/vault_configs/voice_abc/-80/keyring_vault_v2.confsuporte@mysql-01:/opt/vitess/examples/local$ cat /opt/vitess/examples/local/vault_configs/voice_abc/-80/keyring_vault_v2.confvault_url = https://10.128.0.17:8200secret_mount_point = secrets/google/voice_abc/-80secret_mount_point_version = 1token = s.72FVsy2tWics4t6cgl3RML0Mvault_ca = /opt/vitess/examples/local/vault_configs/vault_ca.crtsuporte@mysql-01:/opt/vitess/examples/local$ vim /opt/vitess/examples/local/vault_configs/voice_abc/-80/keyring_vault_v2.confsuporte@mysql-01:/opt/vitess/examples/local$ cat /opt/vitess/examples/local/vault_configs/voice_abc/-80/keyring_vault_v2.confvault_url = https://10.128.0.17:8200secret_mount_point = secrets/google/voice_abc/-80token = s.72FVsy2tWics4t6cgl3RML0Mvault_ca = /opt/vitess/examples/local/vault_configs/vault_ca.crtsuporte@mysql-01:/opt/vitess/examples/local$ xtrabackup --backup --datadir=/vt/vtdataroot/vt_0000000100/data/ --target-dir=/tmp/BKP/ -u xtrabackup_vitess -p -S /vt/vtdataroot/vt_0000000100/mysql.sock --keyring-vault-config=/opt/vitess/examples/local/vault_configs/voice_abc/-80/keyring_vault_v2.confxtrabackup: recognized server arguments: --datadir=/var/lib/mysql --datadir=/vt/vtdataroot/vt_0000000100/data/xtrabackup: recognized client arguments: --backup=1 --target-dir=/tmp/BKP/ --user=xtrabackup_vitess --password --socket=/vt/vtdataroot/vt_0000000100/mysql.sockEnter password:xtrabackup version 8.0.25-17 based on MySQL server 8.0.25 Linux (x86_64) (revision id: d27028b)210910 14:42:35  version_check Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_group=xtrabackup;mysql_socket=/vt/vtdataroot/vt_0000000100/mysql.sock' as 'xtrabackup_vitess'  (using password: YES).210910 14:42:35  version_check Connected to MySQL server210910 14:42:35  version_check Executing a version check against the server...210910 14:42:35  version_check Done.210910 14:42:35 Connecting to MySQL server host: localhost, user: xtrabackup_vitess, password: set, port: not set, socket: /vt/vtdataroot/vt_0000000100/mysql.sockUsing server version 8.0.25-15210910 14:42:35 Executing LOCK TABLES FOR BACKUP...xtrabackup: uses posix_fadvise().xtrabackup: cd to /vt/vtdataroot/vt_0000000100/data/xtrabackup: open files limit requested 0, set to 1024xtrabackup: using the following InnoDB configuration:xtrabackup:   innodb_data_home_dir = /vt/vtdataroot/vt_0000000100/innodb/dataxtrabackup:   innodb_data_file_path = ibdata1:12M:autoextendxtrabackup:   innodb_log_group_home_dir = /vt/vtdataroot/vt_0000000100/innodb/logsxtrabackup:   innodb_log_files_in_group = 2xtrabackup:   innodb_log_file_size = 50331648Number of pools: 1210910 14:42:35 Added plugin 'keyring_vault.so' to load list.2021-09-10T14:42:35.061427Z 0 [System] [MY-011197] [InnoDB] Plugin keyring_vault reported: 'keyring_vault initialization failure. Please check that the keyring_vault_config_file points to readable keyring_vault configuration file. Please also make sure Vault is running and accessible. The keyring_vault will stay unusable until correct configuration file gets provided.'xtrabackup: inititialize_service_handles suceededEncryption can't find master key, please check the keyring is loaded.Cannot read the encryption information in log file header, please check if keyring is loaded.210910 14:42:35 Connecting to MySQL server host: localhost, user: xtrabackup_vitess, password: set, port: not set, socket: /vt/vtdataroot/vt_0000000100/mysql.sockxtrabackup: Redo Log Archiving is not set up.Encryption algorithm support missing: NInnoDB: Assertion failure: redo_log.cc:161:err == DB_SUCCESSInnoDB: thread 140345600612352InnoDB: We intentionally generate a memory trap.InnoDB: Submit a detailed bug report to https://jira.percona.com/projects/PXB.InnoDB: If you get repeated assertion failures or crashes, evenInnoDB: immediately after the mysqld startup, there may beInnoDB: corruption in the InnoDB tablespace. Please refer toInnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.htmlInnoDB: about forcing recovery.14:42:35 UTC - mysqld got signal 6 ;Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.Thread pointer: 0x0Attempting backtrace. You can use the following information to find outwhere mysqld died. If you see no messages after this, something wentterribly wrong...stack_bottom = 0 thread_stack 0x46000xtrabackup(my_print_stacktrace(unsigned char const*, unsigned long)+0x2e) [0x559465afc35e]xtrabackup(handle_fatal_signal+0x31b) [0x5594649e89bb]/lib/x86_64-linux-gnu/libpthread.so.0(+0x12730) [0x7fa4c24da730]/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x10b) [0x7fa4c1af57bb]/lib/x86_64-linux-gnu/libc.so.6(abort+0x121) [0x7fa4c1ae0535]xtrabackup(+0xe2285a) [0x55946444d85a]xtrabackup(+0xefab88) [0x559464525b88]xtrabackup(Redo_Log_Reader::read_logfile(bool, bool*)+0x7f) [0x559464525e7f]xtrabackup(Redo_Log_Data_Manager::copy_once(bool, bool*)+0x3f) [0x559464526b6f]xtrabackup(Redo_Log_Data_Manager::start()+0x9d) [0x559464526d9d]xtrabackup(xtrabackup_backup_func()+0x4e0) [0x5594644e79e0]xtrabackup(main+0x1670) [0x5594644a5c30]/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb) [0x7fa4c1ae209b]xtrabackup(_start+0x2a) [0x5594644d553a]
        Please report a bug at https://jira.percona.com/projects/PXB
        
        

        Link of the new strace using /opt/vitess/examples/local/vault_configs/voice_abc/-80/keyring_vault_v2.conf file:
         
        https://fortics-my.sharepoint.com/:u:/g/personal/benedito_fortics_com_br/EfUaOa7MDGBBnpwoAmEh2_UBRQ7uODBWrW4d8FpI_Ne9_Q?e=cNXick
         
         
         

      Attachments

        Activity

          People

            marcelo.altmann Marcelo Altmann
            benedito BENEDITO MARQUES MAGALHAES
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - Not Specified
                Not Specified
                Logged:
                Time Spent - 3 days, 2 hours
                3d 2h

                Smart Checklist