Uploaded image for project: 'Percona Monitoring and Management'
  1. Percona Monitoring and Management
  2. PMM-3532

Missing mysql db in /var/lib/mysql

    XMLWordPrintable

    Details

      Description

      I'm trying to start docker image in Google Kubernetes.
      It starts fine, but only if external volumes aren't used. Otherwise there are failures: MySQL can't find mysql database in "mounted" directory /var/lib/mysql and starts work only after mysql_install_db executed.

       


      How to reproduce:

      Prepare kubernetes cluster and upload image to Google Cloud repository

      gcloud config set project [PROJECT_NAME]
      gcloud config set compute/zone [ZONE]
      gcloud container clusters create [CLUSTER_NAME]
      gcloud container clusters get-credentials [CLUSTER_NAME]
      export PROJECT_ID="$(gcloud config get-value project -q)"
      gcloud auth configure-docker
      docker pull percona/pmm-server:1.17.0
      docker tag percona/pmm-server:1.17.0 gcr.io/${PROJECT_ID}/pmm-server:1.17.0
      docker push gcr.io/${PROJECT_ID}/pmm-server:1.17.0
      

      Prepare kubernetes deployment

      pod-pmm-server.yaml

      ---
      apiVersion: v1
      kind: Pod
      metadata:
        name: pmm-server
        labels:
          name: pmm-server
      spec:
        containers:
          - name: pmm-server
            image: gcr.io/[PROJECT_NAME]/pmm-server:1.17.0
            ports:
              - containerPort: 80
            volumeMounts:
              - name: pvc-pmm-server-prometheus
                mountPath: /opt/prometheus/data
              - name: pvc-pmm-server-consul
                mountPath: /opt/consul-data
              - name: pvc-pmm-server-mysql
                mountPath: /var/lib/mysql
              - name: pvc-pmm-server-grafana
                mountPath: /var/lib/grafana
            resources:
              requests:
                # cpu: 1
                memory: 1Gi
        volumes:
          - name: pvc-pmm-server-prometheus
            persistentVolumeClaim:
              claimName: pvc-pmm-server-prometheus
          - name: pvc-pmm-server-consul
            persistentVolumeClaim:
              claimName: pvc-pmm-server-consul
          - name: pvc-pmm-server-mysql
            persistentVolumeClaim:
              claimName: pvc-pmm-server-mysql
          - name: pvc-pmm-server-grafana
            persistentVolumeClaim:
              claimName: pvc-pmm-server-grafana
      

      pvc-pmm-server-consul.yaml

      kind: PersistentVolumeClaim
      apiVersion: v1
      metadata:
       name: pvc-pmm-server-consul
      spec:
       accessModes:
       - ReadWriteOnce
       resources:
       requests:
       storage: 20Gi

      pvc-pmm-server-grafana.yaml

      kind: PersistentVolumeClaim
      apiVersion: v1
      metadata:
       name: pvc-pmm-server-grafana
      spec:
       accessModes:
       - ReadWriteOnce
       resources:
       requests:
       storage: 20Gi

      pvc-pmm-server-mysql.yaml

      kind: PersistentVolumeClaim
      apiVersion: v1
      metadata:
       name: pvc-pmm-server-mysql
      spec:
       accessModes:
       - ReadWriteOnce
       resources:
       requests:
       storage: 20Gi

      pvc-pmm-server-prometheus.yaml

      kind: PersistentVolumeClaim
      apiVersion: v1
      metadata:
       name: pvc-pmm-server-prometheus
      spec:
       accessModes:
       - ReadWriteOnce
       resources:
       requests:
       storage: 20Gi
       

      Deploy:

      echo "Create storage"
      kubectl create -f pvc-pmm-server-prometheus.yaml
      kubectl create -f pvc-pmm-server-consul.yaml
      kubectl create -f pvc-pmm-server-grafana.yaml
      kubectl create -f pvc-pmm-server-mysql.yaml
      
      echo "Create pod"
      kubectl create -f pod-pmm-server.yaml
      

      Check it:

      /var/log/entrypoint.sh.log

      + [[ ! 1s =~ ^[1-5]s$ ]]
      + sed s/1s/1s/ /etc/prometheus.yml
      + cat /tmp/prometheus.yml
      + rm -rf /tmp/prometheus.yml
      + sed s/ENV_METRICS_RETENTION/18h/ /etc/supervisord.d/pmm.ini
      + sed -i s/ENV_MAX_CONNECTIONS/15/ /tmp/pmm.ini
      + '[' -n '' ']'
      ++ cat /sys/fs/cgroup/memory/memory.limit_in_bytes
      + MEMORY_LIMIT=9223372036854771712
      ++ awk '{print$2}'
      ++ grep MemTotal /proc/meminfo
      + TOTAL_MEMORY=3878506496
      ++ head -1
      ++ grep -v '^0$'
      ++ sort -n
      ++ printf '%i\n%i\n' 9223372036854771712 3878506496
      + MEMORY_AVAIABLE=3878506496
      + METRICS_MEMORY_MULTIPLIED=541510650
      + [[ 541510650 -lt 134217728 ]]
      + sed -i s/ENV_METRICS_MEMORY_MULTIPLIED/541510650/ /tmp/pmm.ini
      + [[ false = \t\r\u\e ]]
      + cat /tmp/pmm.ini
      + rm -rf /tmp/pmm.ini
      + sed 's/^INTERVAL=.*/INTERVAL=8/' /etc/cron.daily/purge-qan-data
      + cat /tmp/purge-qan-data
      + rm -rf /tmp/purge-qan-data
      + '[' -n Jak-vowt-SIX -a -z '' ']'
      + SERVER_USER=tangoadmin
      + cat
      + pmm-configure -skip-prometheus-reload true -grafana-db-path /var/lib/grafana/grafana.db
      time="2019-02-17T18:26:11Z" level=error msg="Cannot set Grafana password" action=CreateUser error="no such table: user" user=tangoadmin
      time="2019-02-17T18:26:11Z" level=fatal msg="PMM Server is not configured correctly"
      + :
      + '[' -f /var/lib/grafana/grafana.db ']'
      + chown -R pmm:pmm /opt/consul-data
      + chown -R pmm:pmm /opt/prometheus/data
      + chown -R mysql:mysql /var/lib/mysql
      + chown -R grafana:grafana /var/lib/grafana
      + pushd /etc/nginx
      + '[' -s ssl/server.crt ']'
      + '[' -s ssl/server.key ']'
      + '[' -s ssl/dhparam.pem ']'
      + popd
      + '[' -z '' ']'
      + exec supervisord -n -c /etc/supervisord.conf
      2019-02-17 18:26:12,124 CRIT Supervisor running as root (no user in config file)
      2019-02-17 18:26:12,125 WARN Included extra file "/etc/supervisord.d/pmm.ini" during parsing
      2019-02-17 18:26:12,147 INFO RPC interface 'supervisor' initialized
      2019-02-17 18:26:12,148 INFO supervisord started with pid 1
      2019-02-17 18:26:13,151 INFO spawned: 'mysql' with pid 42
      2019-02-17 18:26:13,153 INFO spawned: 'consul' with pid 43
      2019-02-17 18:26:13,165 INFO spawned: 'grafana' with pid 44
      2019-02-17 18:26:13,184 INFO spawned: 'nginx' with pid 45
      2019-02-17 18:26:13,202 INFO spawned: 'cron' with pid 46
      2019-02-17 18:26:13,216 INFO spawned: 'qan-api' with pid 47
      2019-02-17 18:26:13,277 INFO spawned: 'prometheus1' with pid 49
      2019-02-17 18:26:13,300 INFO spawned: 'prometheus' with pid 50
      2019-02-17 18:26:13,324 INFO spawned: 'createdb2' with pid 51
      2019-02-17 18:26:13,347 INFO spawned: 'createdb3' with pid 52
      2019-02-17 18:26:13,381 INFO spawned: 'createdb' with pid 53
      2019-02-17 18:26:13,419 INFO spawned: 'dashboard-upgrade' with pid 55
      2019-02-17 18:26:13,447 INFO spawned: 'node_exporter' with pid 57
      2019-02-17 18:26:13,475 INFO spawned: 'pmm-manage' with pid 59
      2019-02-17 18:26:13,505 INFO spawned: 'pmm-managed' with pid 60
      2019-02-17 18:26:13,909 INFO success: dashboard-upgrade entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
      2019-02-17 18:26:14,162 INFO success: mysql entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
      2019-02-17 18:26:14,162 INFO success: consul entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
      2019-02-17 18:26:14,162 INFO success: grafana entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
      2019-02-17 18:26:14,212 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
      2019-02-17 18:26:14,212 INFO success: cron entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
      2019-02-17 18:26:14,212 INFO success: qan-api entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
      2019-02-17 18:26:14,263 INFO success: prometheus1 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
      2019-02-17 18:26:14,465 INFO success: prometheus entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
      2019-02-17 18:26:14,465 INFO success: createdb2 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
      2019-02-17 18:26:14,465 INFO success: createdb3 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
      2019-02-17 18:26:14,465 INFO success: createdb entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
      2019-02-17 18:26:14,465 INFO success: node_exporter entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
      2019-02-17 18:26:14,465 INFO success: pmm-manage entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
      2019-02-17 18:26:14,544 INFO success: pmm-managed entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
      2019-02-17 18:26:14,676 INFO exited: pmm-managed (exit status 2; expected)
      2019-02-17 18:26:14,746 INFO spawned: 'pmm-managed' with pid 93
      2019-02-17 18:26:14,857 INFO exited: pmm-managed (exit status 2; not expected)
      2019-02-17 18:26:15,347 INFO stopped: grafana (exit status 2)
      2019-02-17 18:26:15,890 INFO spawned: 'pmm-managed' with pid 133
      2019-02-17 18:26:15,908 INFO exited: mysql (exit status 1; not expected)
      2019-02-17 18:26:15,920 INFO spawned: 'mysql' with pid 139
      2019-02-17 18:26:17,658 INFO success: mysql entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
      2019-02-17 18:26:17,658 INFO success: pmm-managed entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
      2019-02-17 18:26:17,677 INFO exited: mysql (exit status 1; not expected)
      2019-02-17 18:26:18,342 INFO spawned: 'mysql' with pid 170
      2019-02-17 18:26:18,355 INFO exited: qan-api (exit status 1; not expected)
      2019-02-17 18:26:18,366 INFO spawned: 'qan-api' with pid 175
      2019-02-17 18:26:18,457 INFO exited: createdb3 (exit status 1; not expected)
      2019-02-17 18:26:18,466 INFO spawned: 'createdb3' with pid 192
      2019-02-17 18:26:18,473 INFO exited: createdb2 (exit status 1; not expected)
      2019-02-17 18:26:18,487 INFO spawned: 'createdb2' with pid 193
      2019-02-17 18:26:18,502 INFO exited: createdb (exit status 1; not expected)
      2019-02-17 18:26:18,540 INFO spawned: 'createdb' with pid 197
      2019-02-17 18:26:20,109 INFO success: mysql entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
      2019-02-17 18:26:20,109 INFO success: qan-api entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
      2019-02-17 18:26:20,109 INFO success: createdb2 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
      2019-02-17 18:26:20,109 INFO success: createdb3 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
      2019-02-17 18:26:20,109 INFO success: createdb entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
      2019-02-17 18:26:20,127 INFO exited: mysql (exit status 1; not expected)
      2019-02-17 18:26:20,679 INFO spawned: 'mysql' with pid 208
      ...
      

      /var/log/mysql.log

       190217 18:26:13 [Note] /usr/sbin/mysqld (mysqld 5.5.61-38.13) starting as process 42 ...
      190217 18:26:14 [Note] Plugin 'FEDERATED' is disabled.
      /usr/sbin/mysqld: Table 'mysql.plugin' doesn't exist
      190217 18:26:14 [ERROR] Can't open the mysql.plugin table. Please run mysql_upgrade to create it.
      190217 18:26:14 InnoDB: The InnoDB memory heap is disabled
      190217 18:26:14 InnoDB: Mutexes and rw_locks use GCC atomic builtins
      190217 18:26:14 InnoDB: Compressed tables use zlib 1.2.7
      190217 18:26:14 InnoDB: Using Linux native AIO
      190217 18:26:14 InnoDB: Initializing buffer pool, size = 128.0M
      190217 18:26:14 InnoDB: Completed initialization of buffer pool
      InnoDB: The first specified data file ./ibdata1 did not exist:
      InnoDB: a new database to be created!
      190217 18:26:14  InnoDB: Setting file ./ibdata1 size to 10 MB
      InnoDB: Database physically writes the file full: wait...
      190217 18:26:14  InnoDB: Log file ./ib_logfile0 did not exist: new to be created
      InnoDB: Setting log file ./ib_logfile0 size to 5 MB
      InnoDB: Database physically writes the file full: wait...
      190217 18:26:14  InnoDB: Log file ./ib_logfile1 did not exist: new to be created
      InnoDB: Setting log file ./ib_logfile1 size to 5 MB
      InnoDB: Database physically writes the file full: wait...
      InnoDB: Doublewrite buffer not found: creating new
      InnoDB: Doublewrite buffer created
      InnoDB: 127 rollback segment(s) active.
      InnoDB: Creating foreign key constraint system tables
      InnoDB: Foreign key constraint system tables created
      190217 18:26:14  InnoDB: Waiting for the background threads to start
      190217 18:26:15 Percona XtraDB (http://www.percona.com) 5.5.61-38.13 started; log sequence number 0
      190217 18:26:15 [ERROR] Fatal error: Can't open and lock privilege tables: Table 'mysql.host' doesn't exist
      190217 18:26:15 [Note] /usr/sbin/mysqld (mysqld 5.5.61-38.13) starting as process 139 ...
      190217 18:26:15 [Note] Plugin 'FEDERATED' is disabled.
      /usr/sbin/mysqld: Table 'mysql.plugin' doesn't exist
      190217 18:26:15 [ERROR] Can't open the mysql.plugin table. Please run mysql_upgrade to create it.
      190217 18:26:15 InnoDB: The InnoDB memory heap is disabled
      190217 18:26:15 InnoDB: Mutexes and rw_locks use GCC atomic builtins
      190217 18:26:15 InnoDB: Compressed tables use zlib 1.2.7
      190217 18:26:15 InnoDB: Using Linux native AIO
      190217 18:26:15 InnoDB: Initializing buffer pool, size = 128.0M
      190217 18:26:16 InnoDB: Completed initialization of buffer pool
      190217 18:26:16 InnoDB: highest supported file format is Barracuda.
      InnoDB: Log scan progressed past the checkpoint lsn 50037
      190217 18:26:16  InnoDB: Database was not shut down normally!
      InnoDB: Starting crash recovery.
      InnoDB: Reading tablespace information from the .ibd files...
      InnoDB: Restoring possible half-written data pages from the doublewrite
      InnoDB: buffer...
      InnoDB: Doing recovery: scanned up to log sequence number 1597945
      190217 18:26:16  InnoDB: Starting an apply batch of log records to the database...
      InnoDB: Progress in percents: 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 
      InnoDB: Apply batch completed
      190217 18:26:16  InnoDB: Waiting for the background threads to start
      190217 18:26:17 Percona XtraDB (http://www.percona.com) 5.5.61-38.13 started; log sequence number 1597945
      190217 18:26:17 [ERROR] Fatal error: Can't open and lock privilege tables: Table 'mysql.host' doesn't exist
      190217 18:26:18 [Note] /usr/sbin/mysqld (mysqld 5.5.61-38.13) starting as process 170 ...
      190217 18:26:18 [Note] Plugin 'FEDERATED' is disabled.
      /usr/sbin/mysqld: Table 'mysql.plugin' doesn't exist
      190217 18:26:18 [ERROR] Can't open the mysql.plugin table. Please run mysql_upgrade to create it.
      190217 18:26:18 InnoDB: The InnoDB memory heap is disabled
      190217 18:26:18 InnoDB: Mutexes and rw_locks use GCC atomic builtins
      190217 18:26:18 InnoDB: Compressed tables use zlib 1.2.7
      190217 18:26:18 InnoDB: Using Linux native AIO
      190217 18:26:18 InnoDB: Initializing buffer pool, size = 128.0M
      190217 18:26:18 InnoDB: Completed initialization of buffer pool
      190217 18:26:18 InnoDB: highest supported file format is Barracuda.
      InnoDB: Log scan progressed past the checkpoint lsn 50037
      190217 18:26:18  InnoDB: Database was not shut down normally!
      InnoDB: Starting crash recovery.
      InnoDB: Reading tablespace information from the .ibd files...
      InnoDB: Restoring possible half-written data pages from the doublewrite
      InnoDB: buffer...
      InnoDB: Doing recovery: scanned up to log sequence number 1597945
      190217 18:26:18  InnoDB: Starting an apply batch of log records to the database...
      InnoDB: Progress in percents: 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 
      InnoDB: Apply batch completed
      190217 18:26:19  InnoDB: Waiting for the background threads to start
      190217 18:26:20 Percona XtraDB (http://www.percona.com) 5.5.61-38.13 started; log sequence number 1597945
      190217 18:26:20 [ERROR] Fatal error: Can't open and lock privilege tables: Table 'mysql.host' doesn't exist
      190217 18:26:20 [Note] /usr/sbin/mysqld (mysqld 5.5.61-38.13) starting as process 208 ...
      190217 18:26:20 [Note] Plugin 'FEDERATED' is disabled.
      /usr/sbin/mysqld: Table 'mysql.plugin' doesn't exist
      190217 18:26:20 [ERROR] Can't open the mysql.plugin table. Please run mysql_upgrade to create it.
      190217 18:26:20 InnoDB: The InnoDB memory heap is disabled
      190217 18:26:20 InnoDB: Mutexes and rw_locks use GCC atomic builtins
      190217 18:26:20 InnoDB: Compressed tables use zlib 1.2.7
      190217 18:26:20 InnoDB: Using Linux native AIO
      190217 18:26:20 InnoDB: Initializing buffer pool, size = 128.0M
      190217 18:26:20 InnoDB: Completed initialization of buffer pool
      190217 18:26:20 InnoDB: highest supported file format is Barracuda.
      InnoDB: Log scan progressed past the checkpoint lsn 50037
      190217 18:26:20  InnoDB: Database was not shut down normally!
      InnoDB: Starting crash recovery.
      InnoDB: Reading tablespace information from the .ibd files...
      InnoDB: Restoring possible half-written data pages from the doublewrite
      InnoDB: buffer...
      InnoDB: Doing recovery: scanned up to log sequence number 1597945
      190217 18:26:20  InnoDB: Starting an apply batch of log records to the database...
      InnoDB: Progress in percents: 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 
      InnoDB: Apply batch completed
      190217 18:26:21  InnoDB: Waiting for the background threads to start
      190217 18:26:22 Percona XtraDB (http://www.percona.com) 5.5.61-38.13 started; log sequence number 1597945
      190217 18:26:22 [ERROR] Fatal error: Can't open and lock privilege tables: Table 'mysql.host' doesn't exist
      ...
      

       How to fix it manually:

      mysql_install_db --datadir=/var/lib/mysql && chown -R mysql:mysql /var/lib/mysql

        Smart Checklist

          Attachments

            Activity

              People

              Assignee:
              Unassigned
              Reporter:
              yusdyr Denis Iusupov
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: