Uploaded image for project: 'Percona XtraDB Cluster'
  1. Percona XtraDB Cluster
  2. PXC-2154

rsync sst shouldn't be used with keyring_vault plugin for encryption


    • Type: Bug
    • Status: Done
    • Priority: Medium
    • Resolution: Fixed
    • Affects Version/s: 5.7.22
    • Fix Version/s: 5.7.22
    • Component/s: None
    • Security Level: Public Bug
    • Labels:


      consider following:
      1. start node1 bootstrapped using keyring_vault plugin and in mysql vault config having specifically every node use its own secret mount point in the vault (so node1 will have something like pxc_node1, node2 pxc_node2 etc.)
      2. enter following in node1:

      create database test;
      use test;
      create table t1 (a int primary key) encryption='Y';
      insert into t1 values (1),(2),(3);

      3. start node2 with empty datadir (using rsync for sst)
      4. observe that it cannot decrypt table t1 because it doesn't have keys to decrypt

      node2> use test;
      Database changed
      node2> show tables;
      | Tables_in_test |
      | t1             |
      1 row in set (0.00 sec)
      node2> select * from t1;
      ERROR 3185 (HY000): Can't find master key from keyring, please check in the server log if a keyring plugin is loaded and initialized successfully.

      The thing is from what I see if we use the same mount point in the vault for secrets (for node2 as for node1) then it will work, but we cannot assume that.
      Maybe best would be to make a note in the docs that rsync just shouldn't be used with keyring_vault plugin because it doesn't work in all cases and discourage it in the scripts if possible but I'll leave it up to you.

      error log info:

      2018-06-28T18:19:48.207963Z 0 [Note] Auto generated RSA key files are placed in data directory.
      2018-06-28T18:19:48.208012Z 0 [Note] Server hostname (bind-address): '*'; port: 19200
      2018-06-28T18:19:48.208026Z 0 [Note] IPv6 is available.
      2018-06-28T18:19:48.208031Z 0 [Note]   - '::' resolves to '::';
      2018-06-28T18:19:48.208039Z 0 [Note] Server socket created on IP: '::'.
      2018-06-28T18:19:48.212533Z 0 [Note] Using encryption for temporary files
      2018-06-28T18:19:48.221178Z 0 [Note] Event Scheduler: Loaded 0 events
      2018-06-28T18:19:48.221241Z 0 [Note] WSREP: Signalling provider to continue on SST completion.
      2018-06-28T18:19:48.221252Z 0 [Note] WSREP: Initialized wsrep sidno 2
      2018-06-28T18:19:48.221262Z 0 [Note] WSREP: SST received: ca5cc0a9-7afd-11e8-9376-82122f737bed:3
      2018-06-28T18:19:48.221310Z 0 [Note] /home/plavi/test/pxc/bin/Percona-XtraDB-Cluster-5.7.22-rel22-29.26.1.Linux.x86_64.ssl100/bin/mysqld: ready for connections.
      Version: '5.7.22-22-29.26'  socket: '/home/plavi/test/pxc/bin/Percona-XtraDB-Cluster-5.7.22-rel22-29.26.1.Linux.x86_64.ssl100/node2/socket.sock'  port: 19200  Percona XtraDB Cluster binary (GPL) 5.7.22-29.26, Revision 9d78ead, wsrep_29.26
      2018-06-28T18:19:48.221505Z 0 [Note] WSREP: 0.0 (bender): State transfer from 1.0 (bender) complete.
      2018-06-28T18:19:48.221523Z 0 [Note] WSREP: SST leaving flow control
      2018-06-28T18:19:48.221525Z 0 [Note] WSREP: Shifting JOINER -> JOINED (TO: 3)
      2018-06-28T18:19:48.221705Z 0 [Note] WSREP: Member 0.0 (bender) synced with group.
      2018-06-28T18:19:48.221726Z 0 [Note] WSREP: Shifting JOINED -> SYNCED (TO: 3)
      2018-06-28T18:19:48.221740Z 4 [Note] WSREP: Synchronized with group, ready for connections
      2018-06-28T18:19:48.221757Z 4 [Note] WSREP: Setting wsrep_ready to true
      2018-06-28T18:19:48.221762Z 4 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
      2018-06-28T18:20:30.142147Z 5 [ERROR] InnoDB: Failed to find tablespace for table `test`.`t1` in the cache. Attempting to load the tablespace with space id 23
      2018-06-28T18:20:30.142261Z 5 [ERROR] InnoDB: Failed to decrpt encryption information, please check key file is not changed!
      2018-06-28T18:20:30.142275Z 5 [ERROR] InnoDB: Encryption information in datafile: ./test/t1.ibd can't be decrypted , please confirm the keyfile is match and keyring plugin is loaded.





              • Assignee:
                krunal.bauskar Krunal Bauskar
                tomislav.plavcic@percona.com Tomislav Plavcic
              • Votes:
                0 Vote for this issue
                2 Start watching this issue


                • Created:

                  Time Tracking

                  Original Estimate - Not Specified
                  Not Specified
                  Remaining Estimate - 0 minutes
                  Time Spent - 4 hours