Uploaded image for project: 'Percona Server for MongoDB'
  1. Percona Server for MongoDB
  2. PSMDB-546

PSMDB upgrade path should use update package instead of remove & install

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Priority: Medium
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Documentation
    • Labels:
      None
    • Needs QA:
      Yes
    • Needs Doc:
      Yes

      Description

      As per the documentation for PSMDB MongoDB upgrade (for all versions), it suggests to remove the existing version package and install the newer version when using the repository for Rhel, Debian etc. But this causes the mongod user/group to be dropped and also the conf file to be saved from /etc/mongod.conf to /etc/mongod.conf.rpmsave 

      When custom path are used for db / log files and in rare case, this method of upgrade causes the data files or log files to lose their mongod permission and leads to error when starting with newer version. The solution is to upgrade package (yum update percona-server-mongodb) instead of remove & install mongodb packages.

      Solution:

      The documentation should be updated to reflect the steps with update instead of remove as follows:

      The current documentation:

      1. Remove Percona Server for MongoDB 4.0 packages: yum remove percona-server-mongodb*

      2. Install Percona Server for MongoDB 4.2 packages: yum install percona-server-mongodb

        {{}}

      The newer suggested way of upgrade path (for example to upgrade to 4.2 from 4.0):

      percona-release enable psmdb-42 release

      yum update percona-server-mongodb

       

      How to repeat:

      Edit conf file with custom dbPath and start mongod service

      [root@app 7]# vi /etc/mongod.conf 
      [root@app 7]# 
      [root@app 7]# grep dbPath /etc/mongod.conf 
      dbPath: /data/mongodb
      [root@app 7]# ls -ld /data/mongodb
      drwxr-xr-x. 2 mongod mongod 4096 Jan 1 14:44 /data/mongodb/
      [root@app 7]# 
      [root@app 7]# grep mongo /etc/passwd
      mongod:x:999:997:mongod:/var/lib/mongo:/bin/false
      [root@app 7]#
      

       

      Start the psmdb 4.0:

      [root@app 7]# service mongod start
      Redirecting to /bin/systemctl start mongod.service
      [root@app 7]# service mongod stop
      Redirecting to /bin/systemctl stop mongod.service
      

      Now remove the MongoDB Package:

      [root@app 7]# yum remove percona-server-mongodb*
      Loaded plugins: fastestmirror
      Resolving Dependencies
      --> Running transaction check
      ---> Package percona-server-mongodb.x86_64 0:4.0.13-7.el7 will be erased
      ---> Package percona-server-mongodb-mongos.x86_64 0:4.0.13-7.el7 will be erased
      ---> Package percona-server-mongodb-server.x86_64 0:4.0.13-7.el7 will be erased
      ---> Package percona-server-mongodb-shell.x86_64 0:4.0.13-7.el7 will be erased
      ....
      ....
      Complete!

      Then check the permission for the data directory and as expected, they look like below - 999:997 which was uid/gid of dropped mongod user/group as part of "yum remove ".

      [root@app 7]# ls -ld /data/mongodb/
      drwxr-xr-x. 4 999 997 4096 Jan 1 15:13 /data/mongodb/
      [root@app 7]#

      Now create a system user/group called testusr as follows and you will be surprised to see the permission of /data/mongodb. This is because the user was created with same uid/gid of dropped mongod user:

      [root@app 7]# useradd -M -r testusr
      [root@app 7]# grep test /etc/passwd
      testusr:x:999:997::/home/testusr:/bin/bash
      [root@app 7]# ls -ld /data/mongodb/
      drwxr-xr-x. 4 testusr testusr 4096 Jan 1 15:13 /data/mongodb/

      Upgrade to PSMDB 4.2 package now:

      [root@app 7]# percona-release enable psmdb-42 release
      * Enabling the Percona Server for MongoDB 4.2 repository
      <*> All done!
      [root@app 7]#
      [root@app 7]# yum install percona-server-mongodb
      Loaded plugins: fastestmirror
      Loading mirror speeds from cached hostfile
       * base: repos-va.psychz.net
       * epel: iad.mirror.rackspace.com
       * extras: repos-va.psychz.net
       * updates: repos-va.psychz.net
      percona-release-noarch | 2.9 kB 00:00:00
      percona-release-x86_64 | 2.9 kB 00:00:00
      psmdb-42-release-noarch | 2.9 kB 00:00:00
      psmdb-42-release-x86_64 | 2.9 kB 00:00:00
      (1/2): psmdb-42-release-noarch/7/primary_db | 1.1 kB 00:00:00
      (2/2): psmdb-42-release-x86_64/7/primary_db | 16 kB 00:00:00
      Resolving Dependencies
      --> Running transaction check
      ---> Package percona-server-mongodb.x86_64 0:4.2.2-3.el7 will be installed
      --> Processing Dependency: percona-server-mongodb-tools = 4.2.2-3.el7 for package: percona-server-mongodb-4.2.2-3.el7.x86_64
      ....
      ....
      ....
      Complete!

      Check the permission for the custom data directory path (/data/mongodb) and to the default path(/var/lib/mongo). We can notice that the mongod permission for the custom path /data/mongodb was not restored by its own. Also the /etc/mongod.conf is moved to /etc/mongod.conf.rpmsave while installing the PSMDB package and new file /etc/mongod.conf is created from the package installation. So need to edit the conf file as needed or restore the old conf file:

      [root@app 7]# ls -ld /data/mongodb/
      drwxr-xr-x. 4 testusr testusr 4096 Jan 1 15:13 /data/mongodb/
      [root@app 7]# ls /etc/mongod*
      /etc/mongod.conf /etc/mongod.conf.rpmsave
      [root@app 7]# vi /etc/mongod.conf

      The following error will be received when starting with the lost permission:

      [root@app 7]# grep dbPath /etc/mongod.conf
       dbPath: /data/mongodb
      [root@app 7]# systemctl start mongod
      Job for mongod.service failed because the control process exited with error code. See "systemctl status mongod.service" and "journalctl -xe" for details.
      [root@app 7]# tail /var/log/mongo/mongod.log 
      2020-01-01T15:23:31.929+0000 I CONTROL [initandlisten] build environment:
      2020-01-01T15:23:31.929+0000 I CONTROL [initandlisten] distarch: x86_64
      2020-01-01T15:23:31.929+0000 I CONTROL [initandlisten] target_arch: x86_64
      2020-01-01T15:23:31.929+0000 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1", port: 27017 }, processManagement: { fork: true, pidFilePath: "/var/run/mongod.pid" }, storage: { dbPath: "/data/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongo/mongod.log" } }
      2020-01-01T15:23:31.930+0000 I STORAGE [initandlisten] exception in initAndListen: IllegalOperation: Attempted to create a lock file on a read-only directory: /data/mongodb, terminating
      2020-01-01T15:23:31.930+0000 I NETWORK [initandlisten] shutdown: going to close listening sockets...
      2020-01-01T15:23:31.930+0000 I NETWORK [initandlisten] removing socket file: /tmp/mongodb-27017.sock
      2020-01-01T15:23:31.930+0000 I - [initandlisten] Stopping further Flow Control ticket acquisitions.
      2020-01-01T15:23:31.930+0000 I CONTROL [initandlisten] now exiting
      2020-01-01T15:23:31.930+0000 I CONTROL [initandlisten] shutting down with code:100

       

        Smart Checklist

          Attachments

            Activity

              People

              Assignee:
              anastasia.alexandrova Anastasia Alexandrova
              Reporter:
              vinodh.krishnaswamy Vinodh Krishnaswamy
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0 minutes
                  0m
                  Logged:
                  Time Spent - 3 days, 3 hours, 15 minutes
                  3d 3h 15m