[PMM-10029] grpc: received message larger than max - Percona JIRA

Details

Type: Bug
Status: Done
Priority: High
Resolution: Fixed
Affects Version/s: 2.27.0
Fix Version/s: 2.33.0
Component/s: None
Labels:
None
Environment:

pmm-server Docker container on Ubuntu 18.04.3 LTS (2vCPU AMD EPYC 7571 8GB RAM EC2 VM Free mem: 1 GB RAM)

Planned Version/s:

2.32.0, 2.33.0
Story Points:
2
Needs Review:
Yes
Needs QA:
Yes
Needs Doc:
Yes
Platform Team:
C/S Core
How to test:

Hide

1. Having a long-running PMM-Instance
2. Upgrade PMM to this FB
3. Now try to Upgrade via UI to another latest version when your step 2 already succeeded

Show
1. Having a long-running PMM-Instance 2. Upgrade PMM to this FB 3. Now try to Upgrade via UI to another latest version when your step 2 already succeeded

Description

Seeing this error every now and then in pmm-managed.log:

ERRO[2022-05-16T11:09:04.928+00:00] rpc error: code = ResourceExhausted desc = grpc: received message larger than max (33565605 vs. 20971520) failed to sent CollectRequest to QAN github.com/percona/pmm-managed/services/qan.(*Client).Collect /home/builder/rpm/BUILD/pmm-managed-ed249a02ab49a3ac0249c60911af1f4d280d5219/src/github.com/percona/pmm-managed/services/qan/client.go:252 github.com/percona/pmm-managed/services/agents.(*Handler).Run.func3 /home/builder/rpm/BUILD/pmm-managed-ed249a02ab49a3ac0249c60911af1f4d280d5219/src/github.com/percona/pmm-managed/services/agents/handler.go:135 runtime/pprof.Do /usr/local/go/src/runtime/pprof/runtime.go:40 github.com/percona/pmm-managed/services/agents.(*Handler).Run /home/builder/rpm/BUILD/pmm-managed-ed249a02ab49a3ac0249c60911af1f4d280d5219/src/github.com/percona/pmm-managed/services/agents/handler.go:134 github.com/percona/pmm-managed/services/agents/grpc.(*agentServer).Connect /home/builder/rpm/BUILD/pmm-managed-ed249a02ab49a3ac0249c60911af1f4d280d5219/src/github.com/percona/pmm-managed/services/agents/grpc/agent_server.go:41 github.com/percona/pmm/api/agentpb._Agent_Connect_Handler /home/builder/go/pkg/mod/github.com/percona/[email protected]/api/agentpb/agent_grpc.pb.go:98 github.com/grpc-ecosystem/go-grpc-middleware/validator.StreamServerInterceptor.func1 /home/builder/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/validator/validator.go:72 github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1 /home/builder/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49 github.com/percona/pmm-managed/utils/interceptors.StreamServiceEnabledInterceptor.func1 /home/builder/rpm/BUILD/pmm-managed-ed249a02ab49a3ac0249c60911af1f4d280d5219/src/github.com/percona/pmm-managed/utils/interceptors/service.go:50 github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1 /home/builder/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49 github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).StreamServerInterceptor.func1 /home/builder/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/server_metrics.go:121 github.com/percona/pmm-managed/utils/interceptors.Stream.func1 /home/builder/rpm/BUILD/pmm-managed-ed249a02ab49a3ac0249c60911af1f4d280d5219/src/github.com/percona/pmm-managed/utils/interceptors/interceptors.go:125 github.com/percona/pmm-managed/utils/interceptors.logRequest /home/builder/rpm/BUILD/pmm-managed-ed249a02ab49a3ac0249c60911af1f4d280d5219/src/github.com/percona/pmm-managed/utils/interceptors/interceptors.go:78 github.com/percona/pmm-managed/utils/interceptors.Stream /home/builder/rpm/BUILD/pmm-managed-ed249a02ab49a3ac0249c60911af1f4d280d5219/src/github.com/percona/pmm-managed/utils/interceptors/interceptors.go:122 github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1.1 /home/builder/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:49 github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1 /home/builder/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/chain.go:58 google.golang.org/grpc.(*Server).processStreamingRPC /home/builder/go/pkg/mod/google.golang.org/[email protected]/server.go:1548 google.golang.org/grpc.(*Server).handleStream /home/builder/go/pkg/mod/google.golang.org/[email protected]/server.go:1623 google.golang.org/grpc.(*Server).serveStreams.func1.2 /home/builder/go/pkg/mod/google.golang.org/[email protected]/server.go:921 runtime.goexit /usr/local/go/src/runtime/asm_amd64.s:1581 agent_id=pmm-server request=6348dc32-d146-11ec-8397-0242ac110002

I also saw this error many times in the web UI when trying to Upgrade to v2.28.0 (browser console showed: "Failed to load resource: the server responded with a status of 429 (Too Many Requests"):

grpc: received message larger than max (15165483 vs. 5242880)

Upgrade log:

ProjectName: pmm-update
Version: 2.27.0
PMMVersion: 2.27.0
Timestamp: 2022-04-12 11:11:07 (UTC)
FullCommit: 3112681ff16267dad7b883601869a8ea7a6381be
Starting "yum --verbose info installed pmm-update" ...
Loading "changelog" plugin
Loading "fastestmirror" plugin
Loading "ovl" plugin
Config time: 0.014
rpmdb time: 0.000
ovl: Copying up (0) files from OverlayFS lower layer
Yum version: 3.4.3
Installed Packages
Name        : pmm-update
Arch        : noarch
Version     : 2.27.0
Release     : 64.2204121111.3112681.el7
Size        : 2.2 M
Repo        : installed
From repo   : pmm2-server
Committer   : Nikita Beletskii <[email protected]>
Committime  : Tue Oct 19 12:00:00 2021
Buildtime   : Tue Apr 12 11:11:09 2022
Install time: Mon Apr 25 09:52:53 2022
Installed by: System <unset>
Changed by  : System <unset>
Summary     : Tool for updating packages and OS configuration for PMM Server
URL         : https://github.com/percona/pmm-update
License     : AGPLv3
Description : Tool for updating packages and OS configuration for PMM ServerStarting "yum update --assumeyes pmm-update" ...
Loaded plugins: changelog, fastestmirror, ovl
Loading mirror speeds from cached hostfile
 * base: download.cf.centos.org
 * epel: d2lzkl7pfhq30w.cloudfront.net
 * extras: download.cf.centos.org
 * updates: download.cf.centos.org
Resolving Dependencies
--> Running transaction check
---> Package pmm-update.noarch 0:2.27.0-64.2204121111.3112681.el7 will be updated
---> Package pmm-update.noarch 0:2.28.0-64.2204291517.0e0bfc6.el7 will be an update
--> Finished Dependency ResolutionDependencies Resolved================================================================================
 Package      Arch     Version                              Repository     Size
================================================================================
Updating:
 pmm-update   noarch   2.28.0-64.2204291517.0e0bfc6.el7     pmm2-server   823 kTransaction Summary
================================================================================
Upgrade  1 PackageTotal download size: 823 k
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating   : pmm-update-2.28.0-64.2204291517.0e0bfc6.el7.noarch           1/2 
  Cleanup    : pmm-update-2.27.0-64.2204121111.3112681.el7.noarch           2/2 
  Verifying  : pmm-update-2.28.0-64.2204291517.0e0bfc6.el7.noarch           1/2 
  Verifying  : pmm-update-2.27.0-64.2204121111.3112681.el7.noarch           2/2 Updated:
  pmm-update.noarch 0:2.28.0-64.2204291517.0e0bfc6.el7                          Complete!
Starting "yum --verbose info installed pmm-update" ...
Loading "changelog" plugin
Loading "fastestmirror" plugin
Loading "ovl" plugin
Config time: 0.016
rpmdb time: 0.000
ovl: Copying up (0) files from OverlayFS lower layer
Yum version: 3.4.3
Installed Packages
Name        : pmm-update
Arch        : noarch
Version     : 2.28.0
Release     : 64.2204291517.0e0bfc6.el7
Size        : 2.2 M
Repo        : installed
From repo   : pmm2-server
Committer   : Nikita Beletskii <[email protected]>
Committime  : Tue Oct 19 12:00:00 2021
Buildtime   : Fri Apr 29 15:17:04 2022
Install time: Mon May 16 11:20:35 2022
Installed by: System <unset>
Changed by  : System <unset>
Summary     : Tool for updating packages and OS configuration for PMM Server
URL         : https://github.com/percona/pmm-update
License     : AGPLv3
Description : Tool for updating packages and OS configuration for PMM Servertime="2022-05-16T11:20:36Z" level=info msg="pmm-update:\nbefore update = {Version:2.27.0 FullVersion:2.27.0-64.2204121111.3112681.el7 BuildTime:2022-04-12 11:11:09 +0000 UTC Repo:pmm2-server}\n after update = {Version:2.28.0 FullVersion:2.28.0-64.2204291517.0e0bfc6.el7 BuildTime:2022-04-29 15:17:04 +0000 UTC Repo:pmm2-server}"
time="2022-05-16T11:20:36Z" level=info msg="Version changed, exiting."
ProjectName: pmm-update
Version: 2.28.0
PMMVersion: 2.28.0
Timestamp: 2022-04-29 15:17:01 (UTC)
FullCommit: 0e0bfc6194bc83292ad4fc6abba869c72b517305
Starting "yum --verbose info installed pmm-update" ...
Loading "changelog" plugin
Loading "fastestmirror" plugin
Loading "ovl" plugin
Config time: 0.015
rpmdb time: 0.000
ovl: Copying up (0) files from OverlayFS lower layer
Yum version: 3.4.3
Installed Packages
Name        : pmm-update
Arch        : noarch
Version     : 2.28.0
Release     : 64.2204291517.0e0bfc6.el7
Size        : 2.2 M
Repo        : installed
From repo   : pmm2-server
Committer   : Nikita Beletskii <[email protected]>
Committime  : Tue Oct 19 12:00:00 2021
Buildtime   : Fri Apr 29 15:17:04 2022
Install time: Mon May 16 11:20:35 2022
Installed by: System <unset>
Changed by  : System <unset>
Summary     : Tool for updating packages and OS configuration for PMM Server
URL         : https://github.com/percona/pmm-update
License     : AGPLv3
Description : Tool for updating packages and OS configuration for PMM ServerStarting "yum update --assumeyes pmm-update" ...
Loaded plugins: changelog, fastestmirror, ovl
Loading mirror speeds from cached hostfile
 * base: download.cf.centos.org
 * epel: d2lzkl7pfhq30w.cloudfront.net
 * extras: download.cf.centos.org
 * updates: download.cf.centos.org
No packages marked for update
Starting "yum --verbose info installed pmm-update" ...
Loading "changelog" plugin
Loading "fastestmirror" plugin
Loading "ovl" plugin
Config time: 0.022
rpmdb time: 0.000
ovl: Copying up (0) files from OverlayFS lower layer
Yum version: 3.4.3
Installed Packages
Name        : pmm-update
Arch        : noarch
Version     : 2.28.0
Release     : 64.2204291517.0e0bfc6.el7
Size        : 2.2 M
Repo        : installed
From repo   : pmm2-server
Committer   : Nikita Beletskii <[email protected]>
Committime  : Tue Oct 19 12:00:00 2021
Buildtime   : Fri Apr 29 15:17:04 2022
Install time: Mon May 16 11:20:35 2022
Installed by: System <unset>
Changed by  : System <unset>
Summary     : Tool for updating packages and OS configuration for PMM Server
URL         : https://github.com/percona/pmm-update
License     : AGPLv3
Description : Tool for updating packages and OS configuration for PMM Servertime="2022-05-16T11:20:39Z" level=info msg="pmm-update:\nbefore update = {Version:2.28.0 FullVersion:2.28.0-64.2204291517.0e0bfc6.el7 BuildTime:2022-04-29 15:17:04 +0000 UTC Repo:pmm2-server}\n after update = {Version:2.28.0 FullVersion:2.28.0-64.2204291517.0e0bfc6.el7 BuildTime:2022-04-29 15:17:04 +0000 UTC Repo:pmm2-server}"
time="2022-05-16T11:20:39Z" level=info msg="Version did not change."
Starting "ansible-playbook --flush-cache /usr/share/pmm-update/ansible/playbook/tasks/update.yml" ...
[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'PLAY [localhost] ***************************************************************TASK [Gathering Facts] *********************************************************
ok: [localhost]TASK [detect /srv/pmm-distribution] ********************************************
ok: [localhost]TASK [detect containers] *******************************************************
ok: [localhost]TASK [force container] *********************************************************
skipping: [localhost]TASK [Remove percona-dashboard without architecture] ***************************
ok: [localhost]TASK [Update percona-dashboards package] ***************************************changed: [localhost]TASK [Create grafana config] ***************************************************TASK [Create grafana directory] ************************************************
ok: [localhost] => (item=/srv/grafana)
changed: [localhost] => (item=/srv/grafana/plugins)TASK [grafana : Set Grafana folder for plugins on /srv partition for all users] ***
ok: [localhost]TASK [Copy new version of grafana.ini] *****************************************
changed: [localhost]TASK [grafana : Create provisioning directory] *********************************
ok: [localhost] => (item=datasources)
ok: [localhost] => (item=plugins)
ok: [localhost] => (item=dashboards)TASK [Copy grafana provisioning files] *****************************************
ok: [localhost] => (item=datasources)
ok: [localhost] => (item=plugins)
ok: [localhost] => (item=dashboards)TASK [Upgrade grafana database (Get the latest schema)] ************************
changed: [localhost]TASK [Supervisord start        | Start supervisord service for AMI/OVF] ********
skipping: [localhost]TASK [Check that supervisor socket exists] *************************************
ok: [localhost]TASK [Supervisord start         | Start supervisord for docker] ****************
skipping: [localhost]TASK [Run initialization playbook] *********************************************TASK [initialization : Get current version] ************************************
ok: [localhost]TASK [initialization : Get image version] **************************************
ok: [localhost]TASK [initialization : Set current version if VERSION fail doen't exist] *******
skipping: [localhost]TASK [initialization : Setting current and image version] **********************
ok: [localhost]TASK [initialization : Setting current and image version] **********************
ok: [localhost]TASK [initialization : Print current version] **********************************
ok: [localhost] => {
    "msg": "Current version: 2.27.0 Image Version: 2.28.0"
}TASK [Checking if we need an update or not] ************************************TASK [dashboards_upgrade : Check that old plugin dir exists] *******************
ok: [localhost]TASK [dashboards_upgrade : Find custom plugin in old plugin dir] ***************
skipping: [localhost]TASK [dashboards_upgrade : Synchronization plugins (old version before 2.22.0)] ***
skipping: [localhost]TASK [dashboards_upgrade : Find custom …