Details
-
Bug
-
Status: Open
-
Low
-
Resolution: Unresolved
-
8.0.26-16 (Q3 2021)
-
None
-
None
Description
Hello Percona
I tried to use hugepages/large_pages with PS8 on Ubuntu 20.04 with apparmor enabled.
/etc/sysctl.conf: entries for 600MB shared memoryvm.nr_hugepages=300
kernel.shmall=153600
vm.hugetlb_shm_group=998
kernel.shmmax=629145600
/etc/mysql/.../...cnf: entries for 256MB innodb buffer
innodb_buffer_pool_size = 256m
large-pages = ON
innodb_buffer_pool_instances = 1
/etc/apparmor.d/local/usr.sbin.mysqld: entries for local path and hugepages ... /{,var/}run/systemd/notify w, capability sys_rawio, capability ipc_lock, capability sys_resource, capability dac_read_search, /proc/*/status r, /etc/ssl/openssl.cnf r
with these entries mysql restart in the mysql error.log: (note 2 lines for each 128MB block of innodb buffer?)
2021-10-26T14:56:34.284043+02:00 0 [Warning] [MY-012678] [InnoDB] Failed to attach shared memory segment, errno 13 2021-10-26T14:56:34.284133+02:00 0 [Warning] [MY-012679] [InnoDB] Using conventional memory pool 2021-10-26T14:56:34.292204+02:00 0 [Warning] [MY-012678] [InnoDB] Failed to attach shared memory segment, errno 13 2021-10-26T14:56:34.292301+02:00 0 [Warning] [MY-012679] [InnoDB] Using conventional memory pool
kern.log: ct 26 15:05:09 db-zzz-115-u kernel: [ 264.933778] audit: type=1400 audit(1635253509.366:21): apparmor="ALLOWED" operation="file_mmap" info="Failed name lookup - disconnected path" error=-13 profile="/usr/sbin/mysqld" name="" pid=10731 comm="mysqld" requested_mask="wr" denied_mask="wr" fsuid=996 ouid=996 Oct 26 15:05:09 db-zzz-115-u kernel: [ 264.941470] audit: type=1400 audit(1635253509.374:22): apparmor="ALLOWED" operation="file_mmap" info="Failed name lookup - disconnected path" error=-13 profile="/usr/sbin/mysqld" name="" pid=10731 comm="mysqld" requested_mask="wr" denied_mask="wr" fsuid=996 ouid=996
the entries were found in various places
- https://man7.org/linux/man-pages/man7/capabilities.7.html -> ipc_lock, sys_rawio as these are used be mmap
- https://bugs.mysql.com/bug.php?id=97768 -> /etc/ssl/openssl.cnf r
-> / rw, but this seems dangerous and did not help
-> ipc_lock - aa-logprof mentioned dac_read_search somewhere while trying to find the problem.
systemctl disable apparmor , reboot and mysql will use hugepages so it is probably an apparmor related problem.
the single hit on google with the kern.log entry (with apparmor="ALLOWED") I could find:
https://lists.ubuntu.com/archives/apparmor/2016-July/009852.html which gives a little description where the
empty name="" in kern.log might come from.
I search the PS8 source tree for the error message:
percona-server-8.0.26-16$ grep -r "Failed to attach shared memory segment"
storage/innobase/os/os0proc.cc: ib::warn(ER_IB_MSG_853) << "Failed to attach shared memory segment,"
which is in these lines:
98 /** Allocates large pages memory. 99 @param[in,out] n Number of bytes to allocate 100 @return allocated memory */ 101 void *os_mem_alloc_large(ulint *n, bool populate) { 102 void *ptr; 103 ulint size; 104 #if defined HAVE_LINUX_LARGE_PAGES && defined UNIV_LINUX 105 int shmid; 106 struct shmid_ds buf; 107 108 if (!os_use_large_pages || !os_large_page_size) { 109 goto skip; 110 } 111 112 /* Align block size to os_large_page_size */ 113 ut_ad(ut_is_2pow(os_large_page_size)); 114 size = ut_2pow_round(*n + (os_large_page_size - 1), os_large_page_size); 115 116 shmid = shmget(IPC_PRIVATE, (size_t)size, SHM_HUGETLB | SHM_R | SHM_W); 117 if (shmid < 0) { 118 ib::warn(ER_IB_MSG_852) 119 << "Failed to allocate " << size << " bytes. errno " << errno; 120 ptr = nullptr; 121 } else { 122 ptr = shmat(shmid, nullptr, 0); 123 if (ptr == (void *)-1) { 124 ib::warn(ER_IB_MSG_853) << "Failed to attach shared memory segment," 125 " errno " 126 << errno; 127 ptr = nullptr; 128 } 129 130 /* Remove the shared memory segment so that it will be 131 automatically freed after memory is detached or 132 process exits */ 133 shmctl(shmid, IPC_RMID, &buf); 134 } 135 136 if (ptr) { 137 *n = size; 138 os_total_large_mem_allocated.fetch_add(size); 139 140 UNIV_MEM_ALLOC(ptr, size); 141 return (ptr); 142 } 143 144 ib::warn(ER_IB_MSG_854) << "Using conventional memory pool"; so shmget in line 116 delivers a shmid >= 0 as it outputs the error in line 124 but shmat does get a -1 What I stumble upon is line 133 as is executed if a shared mem segment was allocated, perhaps this is causing the empty name in the kern.log, but that's wild guessing from my side. (and might cause problems without apparmor if I'm guessing correctly here?)
From one side I guess I might have missed an apparmor setting needed for hugepages, but which?
On the other side apparmor logs ALLOWED in kern.log.