Details
-
Bug
-
Status: Done
-
Medium
-
Resolution: Fixed
-
2.4.x, 8.0.x
-
None
-
None
-
Yes
-
Yes
Description
In short:
PXB still uses Vault API v1, while PS already has support for V2
I'm using Percona Mysql Server 8.0.25 and Xtrabackup 8.0.25 (O.S: Debian 10). The database is in data-at-rest mode, with configuration bellow:
early-plugin-load="keyring_vault=keyring_vault.so" loose-keyring_vault_config="/opt/vitess/examples/local/vault_configs/voice_abc/-80/keyring_vault.conf" binlog_encryption=ON binlog_rotate_encryption_master_key_at_startup=OFF default_table_encryption=ON encrypt_tmp_files=ON innodb_encrypt_online_alter_logs=ON table_encryption_privilege_check=ON innodb_redo_log_encrypt=ON innodb_undo_log_encrypt=ON innodb_temp_tablespace_encrypt=ON
$ cat /opt/vitess/examples/local/vault_configs/voice_abc/-80/keyring_vault.conf
vault_url = https://10.128.0.17:8200
secret_mount_point = secrets/google/voice_abc/-80
secret_mount_point_version = 1
token = xxxxxxxx
vault_ca = /opt/vitess/examples/local/vault_configs/vault_ca.crt
Encryption works fine, but when a try to make a backup using xtrabackup I'm getting error bellow:
suporte@mysql-02:/opt/vitess/examples/local$ xtrabackup --backup --datadir=/vt/vtdataroot/vt_0000000101/data/ --target-dir=/tmp/BKP/ -u xtrabackup_vitess -p -S /vt/vtdataroot/vt_0000000101/mysql.sock --keyring-vault-config=/opt/vitess/examples/local/vault_configs/voice_abc/-80/keyring_vault.conf xtrabackup: recognized server arguments: --datadir=/var/lib/mysql --datadir=/vt/vtdataroot/vt_0000000101/data/ xtrabackup: recognized client arguments: --backup=1 --target-dir=/tmp/BKP/ --user=xtrabackup_vitess --password --socket=/vt/vtdataroot/vt_0000000101/mysql.sock Enter password: xtrabackup version 8.0.25-17 based on MySQL server 8.0.25 Linux (x86_64) (revision id: d27028b) 210909 16:04:34 version_check Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_group=xtrabackup;mysql_socket=/vt/vtdataroot/vt_0000000101/mysql.sock' as 'xtrabackup_vitess' (using password: YES). 210909 16:04:34 version_check Connected to MySQL server 210909 16:04:34 version_check Executing a version check against the server... 210909 16:04:34 version_check Done. 210909 16:04:34 Connecting to MySQL server host: localhost, user: xtrabackup_vitess, password: set, port: not set, socket: /vt/vtdataroot/vt_0000000101/mysql.sock Using server version 8.0.25-15 210909 16:04:34 Executing LOCK TABLES FOR BACKUP... xtrabackup: uses posix_fadvise(). xtrabackup: cd to /vt/vtdataroot/vt_0000000101/data/ xtrabackup: open files limit requested 0, set to 1024 xtrabackup: using the following InnoDB configuration: xtrabackup: innodb_data_home_dir = /vt/vtdataroot/vt_0000000101/innodb/data xtrabackup: innodb_data_file_path = ibdata1:12M:autoextend xtrabackup: innodb_log_group_home_dir = /vt/vtdataroot/vt_0000000101/innodb/logs xtrabackup: innodb_log_files_in_group = 2 xtrabackup: innodb_log_file_size = 50331648 Number of pools: 1 210909 16:04:34 Added plugin 'keyring_vault.so' to load list. 2021-09-09T16:04:34.076080Z 0 [System] [MY-011197] [InnoDB] Plugin keyring_vault reported: 'keyring_vault initialization failure. Please check that the keyring_vault_config_file points to readable keyring_vault configuration file. Please also make sure Vault is running and accessible. The keyring_vault will stay unusable until correct configuration file gets provided.' xtrabackup: inititialize_service_handles suceeded Encryption can't find master key, please check the keyring is loaded. Cannot read the encryption information in log file header, please check if keyring is loaded. 210909 16:04:34 Connecting to MySQL server host: localhost, user: xtrabackup_vitess, password: set, port: not set, socket: /vt/vtdataroot/vt_0000000101/mysql.sock xtrabackup: Redo Log Archiving is not set up. xtrabackup: Can't create/write to file '/tmp/BKP/xtrabackup_logfile' (OS errno 17 - File exists) xtrabackup: error: failed to open the target stream for 'xtrabackup_logfile'. suporte@mysql-02:/opt/vitess/examples/local$
Vault is working properly and is accessible, and configuration file is readable. I try too specify the pluguin directore of percona xtrabackup, but the same error occurs:
suporte@mysql-02:/opt/vitess/examples/local$ sudo dpkg -L percona-xtrabackup-80 | grep keyring_vault.so /usr/lib/xtrabackup/plugin/keyring_vault.so suporte@mysql-02:/opt/vitess/examples/local$ locate keyring_vault.so /usr/lib/mysql/plugin/keyring_vault.so /usr/lib/mysql/plugin/debug/keyring_vault.so /usr/lib/xtrabackup/plugin/keyring_vault.so suporte@mysql-02:/opt/vitess/examples/local$ ls -l /tmp/BKP/ total 0 suporte@mysql-02:/opt/vitess/examples/local$ xtrabackup --backup --datadir=/vt/vtdataroot/vt_0000000101/data/ --target-dir=/tmp/BKP/ -u xtrabackup_vitess -p -S /vt/vtdataroot/vt_0000000101/mysql.sock --keyring-vault-config=/opt/vitess/examples/local/vault_configs/voice_abc/-80/keyring_vault.conf --xtrabackup-plugin-dir=/usr/lib/xtrabackup/plugin/ xtrabackup: recognized server arguments: --datadir=/var/lib/mysql --datadir=/vt/vtdataroot/vt_0000000101/data/ xtrabackup: recognized client arguments: --backup=1 --target-dir=/tmp/BKP/ --user=xtrabackup_vitess --password --socket=/vt/vtdataroot/vt_0000000101/mysql.sock --xtrabackup-plugin-dir=/usr/lib/xtrabackup/plugin/ Enter password: xtrabackup version 8.0.25-17 based on MySQL server 8.0.25 Linux (x86_64) (revision id: d27028b) 210909 17:13:27 version_check Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_group=xtrabackup;mysql_socket=/vt/vtdataroot/vt_0000000101/mysql.sock' as 'xtrabackup_vitess' (using password: YES). 210909 17:13:27 version_check Connected to MySQL server 210909 17:13:27 version_check Executing a version check against the server... 210909 17:13:27 version_check Done. 210909 17:13:27 Connecting to MySQL server host: localhost, user: xtrabackup_vitess, password: set, port: not set, socket: /vt/vtdataroot/vt_0000000101/mysql.sock Using server version 8.0.25-15 210909 17:13:27 Executing LOCK TABLES FOR BACKUP... xtrabackup: uses posix_fadvise(). xtrabackup: cd to /vt/vtdataroot/vt_0000000101/data/ xtrabackup: open files limit requested 0, set to 1024 xtrabackup: using the following InnoDB configuration: xtrabackup: innodb_data_home_dir = /vt/vtdataroot/vt_0000000101/innodb/data xtrabackup: innodb_data_file_path = ibdata1:12M:autoextend xtrabackup: innodb_log_group_home_dir = /vt/vtdataroot/vt_0000000101/innodb/logs xtrabackup: innodb_log_files_in_group = 2 xtrabackup: innodb_log_file_size = 50331648 Number of pools: 1 210909 17:13:27 Added plugin 'keyring_vault.so' to load list. 2021-09-09T17:13:27.695838Z 0 [System] [MY-011197] [InnoDB] Plugin keyring_vault reported: 'keyring_vault initialization failure. Please check that the keyring_vault_config_file points to readable keyring_vault configuration file. Please also make sure Vault is running and accessible. The keyring_vault will stay unusable until correct configuration file gets provided.' xtrabackup: inititialize_service_handles suceeded Encryption can't find master key, please check the keyring is loaded. Cannot read the encryption information in log file header, please check if keyring is loaded. 210909 17:13:27 Connecting to MySQL server host: localhost, user: xtrabackup_vitess, password: set, port: not set, socket: /vt/vtdataroot/vt_0000000101/mysql.sock xtrabackup: Redo Log Archiving is not set up. Encryption algorithm support missing: N InnoDB: Assertion failure: redo_log.cc:161:err == DB_SUCCESS InnoDB: thread 139901056161792InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to https://jira.percona.com/projects/PXB. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 17:13:27 UTC - mysqld got signal 6 ; Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware. Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0 thread_stack 0x46000 xtrabackup(my_print_stacktrace(unsigned char const*, unsigned long)+0x2e) [0x558b0e9d335e] xtrabackup(handle_fatal_signal+0x31b) [0x558b0d8bf9bb] /lib/x86_64-linux-gnu/libpthread.so.0(+0x12730) [0x7f3d4163a730] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x10b) [0x7f3d40c557bb] /lib/x86_64-linux-gnu/libc.so.6(abort+0x121) [0x7f3d40c40535] xtrabackup(+0xe2285a) [0x558b0d32485a] xtrabackup(+0xefab88) [0x558b0d3fcb88] xtrabackup(Redo_Log_Reader::read_logfile(bool, bool*)+0x7f) [0x558b0d3fce7f] xtrabackup(Redo_Log_Data_Manager::copy_once(bool, bool*)+0x3f) [0x558b0d3fdb6f] xtrabackup(Redo_Log_Data_Manager::start()+0x9d) [0x558b0d3fdd9d] xtrabackup(xtrabackup_backup_func()+0x4e0) [0x558b0d3be9e0] xtrabackup(main+0x1670) [0x558b0d37cc30] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb) [0x7f3d40c4209b] xtrabackup(_start+0x2a) [0x558b0d3ac53a] Please report a bug at https://jira.percona.com/projects/PXB suporte@mysql-02:/opt/vitess/examples/local$
This is the link of an strace:
https://fortics-my.sharepoint.com/:u:/g/personal/benedito_fortics_com_br/Ec0e_f8983JIqg3tFIKkXUMBhfUOAUzkFD5srqneRok3Mw?e=bxwjYz
After that, I've been trying to follow the steps bellow, suggested on percona community:
- Copy /opt/vitess/examples/local/vault_configs/voice_abc/-80/keyring_vault.conf to a different file
- Edit the nes file and remove option secret_mount_point_version
- Take the backup again passing the new vault.cnf file.
Same error occurs:
suporte@mysql-01:/opt/vitess/examples/local$ cp /opt/vitess/examples/local/vault_configs/voice_abc/-80/keyring_vault.conf /opt/vitess/examples/local/vault_configs/voice_abc/-80/keyring_vault_v2.confsuporte@mysql-01:/opt/vitess/examples/local$ cat /opt/vitess/examples/local/vault_configs/voice_abc/-80/keyring_vault_v2.confvault_url = https://10.128.0.17:8200secret_mount_point = secrets/google/voice_abc/-80secret_mount_point_version = 1token = s.72FVsy2tWics4t6cgl3RML0Mvault_ca = /opt/vitess/examples/local/vault_configs/vault_ca.crtsuporte@mysql-01:/opt/vitess/examples/local$ vim /opt/vitess/examples/local/vault_configs/voice_abc/-80/keyring_vault_v2.confsuporte@mysql-01:/opt/vitess/examples/local$ cat /opt/vitess/examples/local/vault_configs/voice_abc/-80/keyring_vault_v2.confvault_url = https://10.128.0.17:8200secret_mount_point = secrets/google/voice_abc/-80token = s.72FVsy2tWics4t6cgl3RML0Mvault_ca = /opt/vitess/examples/local/vault_configs/vault_ca.crtsuporte@mysql-01:/opt/vitess/examples/local$ xtrabackup --backup --datadir=/vt/vtdataroot/vt_0000000100/data/ --target-dir=/tmp/BKP/ -u xtrabackup_vitess -p -S /vt/vtdataroot/vt_0000000100/mysql.sock --keyring-vault-config=/opt/vitess/examples/local/vault_configs/voice_abc/-80/keyring_vault_v2.confxtrabackup: recognized server arguments: --datadir=/var/lib/mysql --datadir=/vt/vtdataroot/vt_0000000100/data/xtrabackup: recognized client arguments: --backup=1 --target-dir=/tmp/BKP/ --user=xtrabackup_vitess --password --socket=/vt/vtdataroot/vt_0000000100/mysql.sockEnter password:xtrabackup version 8.0.25-17 based on MySQL server 8.0.25 Linux (x86_64) (revision id: d27028b)210910 14:42:35 version_check Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_group=xtrabackup;mysql_socket=/vt/vtdataroot/vt_0000000100/mysql.sock' as 'xtrabackup_vitess' (using password: YES).210910 14:42:35 version_check Connected to MySQL server210910 14:42:35 version_check Executing a version check against the server...210910 14:42:35 version_check Done.210910 14:42:35 Connecting to MySQL server host: localhost, user: xtrabackup_vitess, password: set, port: not set, socket: /vt/vtdataroot/vt_0000000100/mysql.sockUsing server version 8.0.25-15210910 14:42:35 Executing LOCK TABLES FOR BACKUP...xtrabackup: uses posix_fadvise().xtrabackup: cd to /vt/vtdataroot/vt_0000000100/data/xtrabackup: open files limit requested 0, set to 1024xtrabackup: using the following InnoDB configuration:xtrabackup: innodb_data_home_dir = /vt/vtdataroot/vt_0000000100/innodb/dataxtrabackup: innodb_data_file_path = ibdata1:12M:autoextendxtrabackup: innodb_log_group_home_dir = /vt/vtdataroot/vt_0000000100/innodb/logsxtrabackup: innodb_log_files_in_group = 2xtrabackup: innodb_log_file_size = 50331648Number of pools: 1210910 14:42:35 Added plugin 'keyring_vault.so' to load list.2021-09-10T14:42:35.061427Z 0 [System] [MY-011197] [InnoDB] Plugin keyring_vault reported: 'keyring_vault initialization failure. Please check that the keyring_vault_config_file points to readable keyring_vault configuration file. Please also make sure Vault is running and accessible. The keyring_vault will stay unusable until correct configuration file gets provided.'xtrabackup: inititialize_service_handles suceededEncryption can't find master key, please check the keyring is loaded.Cannot read the encryption information in log file header, please check if keyring is loaded.210910 14:42:35 Connecting to MySQL server host: localhost, user: xtrabackup_vitess, password: set, port: not set, socket: /vt/vtdataroot/vt_0000000100/mysql.sockxtrabackup: Redo Log Archiving is not set up.Encryption algorithm support missing: NInnoDB: Assertion failure: redo_log.cc:161:err == DB_SUCCESSInnoDB: thread 140345600612352InnoDB: We intentionally generate a memory trap.InnoDB: Submit a detailed bug report to https://jira.percona.com/projects/PXB.InnoDB: If you get repeated assertion failures or crashes, evenInnoDB: immediately after the mysqld startup, there may beInnoDB: corruption in the InnoDB tablespace. Please refer toInnoDB: http://dev.mysql.com/doc/refman/8.0/en/forcing-innodb-recovery.htmlInnoDB: about forcing recovery.14:42:35 UTC - mysqld got signal 6 ;Most likely, you have hit a bug, but this error can also be caused by malfunctioning hardware.Thread pointer: 0x0Attempting backtrace. You can use the following information to find outwhere mysqld died. If you see no messages after this, something wentterribly wrong...stack_bottom = 0 thread_stack 0x46000xtrabackup(my_print_stacktrace(unsigned char const*, unsigned long)+0x2e) [0x559465afc35e]xtrabackup(handle_fatal_signal+0x31b) [0x5594649e89bb]/lib/x86_64-linux-gnu/libpthread.so.0(+0x12730) [0x7fa4c24da730]/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x10b) [0x7fa4c1af57bb]/lib/x86_64-linux-gnu/libc.so.6(abort+0x121) [0x7fa4c1ae0535]xtrabackup(+0xe2285a) [0x55946444d85a]xtrabackup(+0xefab88) [0x559464525b88]xtrabackup(Redo_Log_Reader::read_logfile(bool, bool*)+0x7f) [0x559464525e7f]xtrabackup(Redo_Log_Data_Manager::copy_once(bool, bool*)+0x3f) [0x559464526b6f]xtrabackup(Redo_Log_Data_Manager::start()+0x9d) [0x559464526d9d]xtrabackup(xtrabackup_backup_func()+0x4e0) [0x5594644e79e0]xtrabackup(main+0x1670) [0x5594644a5c30]/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb) [0x7fa4c1ae209b]xtrabackup(_start+0x2a) [0x5594644d553a] Please report a bug at https://jira.percona.com/projects/PXB
Link of the new strace using /opt/vitess/examples/local/vault_configs/voice_abc/-80/keyring_vault_v2.conf file:
https://fortics-my.sharepoint.com/:u:/g/personal/benedito_fortics_com_br/EfUaOa7MDGBBnpwoAmEh2_UBRQ7uODBWrW4d8FpI_Ne9_Q?e=cNXick