PXB gets stuck during incremental backup prepare of encrypted tables

Problem:
With pagetracking, PXB crashes during prepare of incremental backup if alter
encryption tablespace='y' happens after the checkpoint.

Analysis:
If modifed pages due to change in encryption are not checkpoint, then the list
of those pages would not be available with the page tracking, and it relies on
redo to apply those changes to the tablespace. The redo apply on page 0 reset
the in-memory encryption, and because of that, remaining pages could not be
decrypted as it doesn't have encryption information.

Fix:
ALTER tablespace encryption='y' DDL forcefully flushes all the pages to disk at
the end. If this alter happens after the checkpoint, PXB scans complete ibd to
know the modified pages. So it doesn't need to apply redo on page 0 as it has
copied the page

Hi , The backup directory before prepare is available on twinb02 machine.

Hey  ,  can you provide me backup directory  on QA03 where prepare fails. I would need an original copy of backup directory so modify your scripts to save backup before prepare.
 
 
 
 

Reproduced the issue with PXB8.0.27(percona-xtrabackup-8.0.27-19-Linux-x86_64.glibc2.17) and PS8.0.27(Percona-Server-8.0.27-18-Linux.x86_64.glibc2.17-debug) release tarballs. PS8.0.27 is running with keyring_file component encryption and PXB8.0.27 is using page tracking.
PS8.0.27 is running as:

PXB process is stuck as

PXB gets stuck during the prepare of second incremental backup and displays the logs

Logs including stack trace of all threads is attached as pxb_2681_stuck_pg_encrypt.tar.gz