1. start node1 bootstrapped using keyring_vault plugin and in mysql vault config having specifically every node use its own secret mount point in the vault (so node1 will have something like pxc_node1, node2 pxc_node2 etc.)
2. enter following in node1:
create database test;
create table t1 (a int primary key) encryption='Y';
insert into t1 values (1),(2),(3);
3. start node2 with empty datadir (using rsync for sst)
4. observe that it cannot decrypt table t1 because it doesn't have keys to decrypt
node2> use test;
node2> show tables;
| Tables_in_test |
| t1 |
1 row in set (0.00 sec)
node2> select * from t1;
ERROR 3185 (HY000): Can't find master key from keyring, please check in the server log if a keyring plugin is loaded and initialized successfully.
The thing is from what I see if we use the same mount point in the vault for secrets (for node2 as for node1) then it will work, but we cannot assume that.
Maybe best would be to make a note in the docs that rsync just shouldn't be used with keyring_vault plugin because it doesn't work in all cases and discourage it in the scripts if possible but I'll leave it up to you.
error log info:
2018-06-28T18:19:48.207963Z 0 [Note] Auto generated RSA key files are placed in data directory.
2018-06-28T18:19:48.208012Z 0 [Note] Server hostname (bind-address): '*'; port: 19200
2018-06-28T18:19:48.208026Z 0 [Note] IPv6 is available.
2018-06-28T18:19:48.208031Z 0 [Note] - '::' resolves to '::';
2018-06-28T18:19:48.208039Z 0 [Note] Server socket created on IP: '::'.
2018-06-28T18:19:48.212533Z 0 [Note] Using encryption for temporary files
2018-06-28T18:19:48.221178Z 0 [Note] Event Scheduler: Loaded 0 events
2018-06-28T18:19:48.221241Z 0 [Note] WSREP: Signalling provider to continue on SST completion.
2018-06-28T18:19:48.221252Z 0 [Note] WSREP: Initialized wsrep sidno 2
2018-06-28T18:19:48.221262Z 0 [Note] WSREP: SST received: ca5cc0a9-7afd-11e8-9376-82122f737bed:3
2018-06-28T18:19:48.221310Z 0 [Note] /home/plavi/test/pxc/bin/Percona-XtraDB-Cluster-5.7.22-rel22-29.26.1.Linux.x86_64.ssl100/bin/mysqld: ready for connections.
Version: '5.7.22-22-29.26' socket: '/home/plavi/test/pxc/bin/Percona-XtraDB-Cluster-5.7.22-rel22-29.26.1.Linux.x86_64.ssl100/node2/socket.sock' port: 19200 Percona XtraDB Cluster binary (GPL) 5.7.22-29.26, Revision 9d78ead, wsrep_29.26
2018-06-28T18:19:48.221505Z 0 [Note] WSREP: 0.0 (bender): State transfer from 1.0 (bender) complete.
2018-06-28T18:19:48.221523Z 0 [Note] WSREP: SST leaving flow control
2018-06-28T18:19:48.221525Z 0 [Note] WSREP: Shifting JOINER -> JOINED (TO: 3)
2018-06-28T18:19:48.221705Z 0 [Note] WSREP: Member 0.0 (bender) synced with group.
2018-06-28T18:19:48.221726Z 0 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 3)
2018-06-28T18:19:48.221740Z 4 [Note] WSREP: Synchronized with group, ready for connections
2018-06-28T18:19:48.221757Z 4 [Note] WSREP: Setting wsrep_ready to true
2018-06-28T18:19:48.221762Z 4 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
2018-06-28T18:20:30.142147Z 5 [ERROR] InnoDB: Failed to find tablespace for table `test`.`t1` in the cache. Attempting to load the tablespace with space id 23
2018-06-28T18:20:30.142261Z 5 [ERROR] InnoDB: Failed to decrpt encryption information, please check key file is not changed!
2018-06-28T18:20:30.142275Z 5 [ERROR] InnoDB: Encryption information in datafile: ./test/t1.ibd can't be decrypted , please confirm the keyfile is match and keyring plugin is loaded.