Uploaded image for project: 'Percona Monitoring and Management'
  1. Percona Monitoring and Management
  2. PMM-8548

AMI upgrade from 2.16.0 and 2.17.0 to 2.20.0 failed

Details

    • Bug
    • Status: Done
    • High
    • Resolution: Fixed
    • 2.20.0
    • None
    • AMI
    • None
    • C/S Core

    Description

      User impact: User can't upgrade PMM

      STR:

      1. Run 2.16.0 AMI PMM
      2. Open Home dashboard and start upgrade to 2.20.0

      Actual Result: Upgrade stuck in loop

       TASK [Cleanup yum metadata] ****************************************************
      skipping: [localhost]
      TASK [Download pmm2 packages] **************************************************
      fatal: [localhost]: FAILED! => {"changed": true, "changes": {"installed": [], "updated": [["pmm-server", "2.20.0-23.2107271100.e68715f.el7.noarch from pmm2-server"], ["percona-victoriametrics", "1.60.0-1.el7.x86_64 from pmm2-server"], ["percona-qan-api2", "2.20.0-15.2107120744.af14526.el7.x86_64 from pmm2-server"], ["dbaas-controller", "2.20.0-1.2106281154.316d3d2.el7.x86_64 from pmm2-server"], ["pmm-managed", "2.20.0-17.2107220951.3c660c3.el7.x86_64 from pmm2-server"], ["percona-grafana", "7.5.7-93.2107220953.908ff0c.el7.x86_64 from pmm2-server"], ["percona-dashboards", "2.20.0-15.2107131605.75d67a4.el7.noarch from pmm2-server"], ["percona-alertmanager", "0.21.0-3.el7.x86_64 from pmm2-server"], ["pmm2-client", "2.20.0-6.el7.x86_64 from percona-release-x86_64"]]}, "msg": "https://repo.percona.com/pmm2-components/yum/release/7/RPMS/x86_64/percona-victoriametrics-1.60.0-1.el7.x86_64.rpm: [Errno -1] Package does not match intended download. Suggestion: run yum --enablerepo=pmm2-server clean metadata\nTrying other mirror.\n\n\nError downloading packages:\n  percona-victoriametrics-1.60.0-1.el7.x86_64: [Errno 256] No more mirrors to try.\n\n", "rc": 1, "results": ["All packages providing pmm-update are up to date", "All packages providing percona-clickhouse-client are up to date", "All packages providing percona-clickhouse-server are up to date", "All packages providing dbaas-tools are up to date", "Loaded plugins: changelog, fastestmirror\nLoading mirror speeds from cached hostfile\n * base: d36uatko69830t.cloudfront.net\n * epel: dl.fedoraproject.org\n * extras: d36uatko69830t.cloudfront.net\n * updates: d36uatko69830t.cloudfront.net\nPackage pmm-update-2.20.0-60.2107271101.d72f0db.el7.noarch already installed and latest version\nPackage percona-clickhouse-client-19.7.5.27-1.el7.x86_64 already installed and latest version\nPackage percona-clickhouse-server-19.7.5.27-1.el7.x86_64 already installed and latest version\nPackage dbaas-tools-0.5.1-4.2009041503.el7.x86_64 already installed and latest version\nResolving Dependencies\n--> Running transaction check\n---> Package dbaas-controller.x86_64 0:2.17.0-1.2105070916.4fd8e4d.el7 will be updated\n---> Package dbaas-controller.x86_64 0:2.20.0-1.2106281154.316d3d2.el7 will be an update\n---> Package percona-alertmanager.x86_64 0:0.20.0-3.el7 will be updated\n---> Package percona-alertmanager.x86_64 0:0.21.0-3.el7 will be an update\n---> Package percona-dashboards.noarch 0:2.17.0-15.2105041522.8f48f1e.el7 will be updated\n---> Package percona-dashboards.noarch 0:2.20.0-15.2107131605.75d67a4.el7 will be an update\n---> Package percona-grafana.x86_64 0:7.3.7-92.2105051427.9d10fa2.el7 will be updated\n---> Package percona-grafana.x86_64 0:7.5.7-93.2107220953.908ff0c.el7 will be an update\n---> Package percona-qan-api2.x86_64 0:2.17.0-15.2104262202.f8aa0a4.el7 will be updated\n---> Package percona-qan-api2.x86_64 0:2.20.0-15.2107120744.af14526.el7 will be an update\n---> Package percona-victoriametrics.x86_64 0:1.53.1-1.el7 will be updated\n---> Package percona-victoriametrics.x86_64 0:1.60.0-1.el7 will be an update\n---> Package pmm-managed.x86_64 0:2.17.0-17.2105070914.e66288f.el7 will be updated\n---> Package pmm-managed.x86_64 0:2.20.0-17.2107220951.3c660c3.el7 will be an update\n---> Package pmm-server.noarch 0:2.17.0-22.2105110659.89e2758.el7 will be updated\n---> Package pmm-server.noarch 0:2.20.0-23.2107271100.e68715f.el7 will be an update\n---> Package pmm2-client.x86_64 0:2.17.0-6.el7 will be updated\n---> Package pmm2-client.x86_64 0:2.20.0-6.el7 will be an update\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package              Arch   Version                          Repository   Size\n================================================================================\nUpdating:\n dbaas-controller     x86_64 2.20.0-1.2106281154.316d3d2.el7  pmm2-server 4.3 M\n percona-alertmanager x86_64 0.21.0-3.el7                     pmm2-server  12 M\n percona-dashboards   noarch 2.20.0-15.2107131605.75d67a4.el7 pmm2-server  32 M\n percona-grafana      x86_64 7.5.7-93.2107220953.908ff0c.el7  pmm2-server  41 M\n percona-qan-api2     x86_64 2.20.0-15.2107120744.af14526.el7 pmm2-server 4.4 M\n percona-victoriametrics\n                      x86_64 1.60.0-1.el7                     pmm2-server 7.9 M\n pmm-managed          x86_64 2.20.0-17.2107220951.3c660c3.el7 pmm2-server  16 M\n pmm-server           noarch 2.20.0-23.2107271100.e68715f.el7 pmm2-server 2.0 M\n pmm2-client          x86_64 2.20.0-6.el7                     percona-release-x86_64\n                                                                           43 M\n\nTransaction Summary\n================================================================================\nUpgrade  9 Packages\n\nTotal size: 161 M\nTotal download size: 7.9 M\nBackground downloading packages, then exiting:\nDelta RPMs disabled because /usr/bin/applydeltarpm not installed.\n"]}
      PLAY RECAP *********************************************************************
      localhost                  : ok=10   changed=0    unreachable=0    failed=1    skipped=2    rescued=0    ignored=0   
      time="2021-08-04T07:57:47Z" level=fatal msg="RunPlaybook failed: exit status 2"
      ProjectName: pmm-update
      Version: 2.20.0
      PMMVersion: 2.20.0
      

      How to test:

      1. Run 2.16.0 AMI PMM
      2. Open Home dashboard and start an upgrade to 2.20.0

      Problem and solution:

      We have the victoriametrics package:
      https://repo.percona.com/pmm2-components/yum/release/7/RPMS/x86_64/percona-victoriametrics-1.60.0-1.el7.x86_64.rpm
      We upload this package on each release. Even if the version didn’t change.
      It works good with testing and experimental repo but for some unknown reason, this breaks the production repo. We’ll stop upload the same package in the next release.

      Expected Result:

      Upgrades from 2.18.0 and 2.19.0 look ok

      I checked upgrade with enabled testing repositories and it was ok

      Attachments

        Issue Links

          Activity

            People

              nikita.beletskii Nikita Beletskii (Inactive)
              nailya.kutlubaeva Nailya Kutlubaeva
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - Not Specified
                  Not Specified
                  Logged:
                  Time Spent - 1 hour
                  1h

                  Smart Checklist