Uploaded image for project: 'Percona Monitoring and Management'
  1. Percona Monitoring and Management
  2. PMM-5682

Increase the limit on shared dashboard size



    • Type: Bug
    • Status: Open
    • Priority: High
    • Resolution: Unresolved
    • Affects Version/s: 2.4.0
    • Fix Version/s: None
    • Component/s: PMM Server
    • Sprint:
      Platform Sprint 16
    • Needs Review:
    • Needs QA:
    • Needs Doc:



      1. Add a few MySQL services(4-5) to monitoring
      2. Open MySQL Compare
      3. Select all service names
      4. Expand collapsed chart's sections
      5. Click on Share, select Snapshot tab
      6. Change Timeout to 150s
      7. Click on Share with Percona

      Given result: The snapshot link generation is failed with the next error:

      Original report
      There's currently an implicit limit on the possible size of dashboard being shared with Percona from PMM. The limit is 1MB. Sharing of dashboards is a feature of Grafana, and as explained in https://github.com/grafana/grafana/issues/9835 the limit is tied to client_max_body_size setting in involved nginx servers.

      There are two sides to this problem: "client" (local PMM installation) and "server" (Percona Snapshot-sharing Grafana). client_max_body_size should be increased from default 1MB to some higher value (10MB was discussed) universally.

      1. On PMM installation side, with default client_max_body_size sharing a dashboard with size larger than 1MB results in the following error logged:

      nginx log
      2020/03/30 12:35:47 [error] 47#47: *8350 client intended to send too large body: 1161812 bytes, client: xxx.xxx.xxx.xxx, server: _, request: "POST /graph/api/snapshots HTTP/2.0", host: "***", referrer: "***://xxx/graph/d/mysql-innodb/mysql-innodb-details?orgId=1&from=1584158400000&to=1584208799000&var-interval=$__auto_interval_interval&var-node_name=***&var-crop_host=***&var-service_name=***-mysql&var-version=5.7.28-31-31.41-log&var-crop_service=***-my&var-environment=All&var-cluster=All&var-replication_set=All&var-database=All"

      2. With the above worked around by adding client_max_body_size 2m; to the installation's nginx, snapshot sharing is failing on Percona side's nginx, with the following being logged on PMM side grafana.log:

      grafana log
      t=2020-03-30T12:39:01+0000 lvl=eror msg="Failed to create external snaphost" logger=context userId=2 orgId=1 uname=*** error="Create external snapshot response status code 413"
      t=2020-03-30T12:39:01+0000 lvl=eror msg="Request Completed" logger=context userId=2 orgId=1 uname=*** method=POST path=/api/snapshots status=500 remote_addr= time_ms=435 size=76 referer="https://***/graph/d/mysql-innodb/mysql-innodb-details?orgId=1&from=1584158400000&to=1584208799000&var-interval=$__auto_interval_interval&var-node_name=***&var-crop_host=***&var-service_name=***-mysql&var-version=5.7.28-31-31.41-log&var-crop_service=***-my&var-environment=All&var-cluster=All&var-replication_set=All&var-database=All"

      To reproduce "local" issue (point 1), the easiest way to go is to add client_max_body_size 1k;, so that sharing almost any dashboard will fail. For point 2, sufficiently large dashboard will have to be shared.

      Existing workaround is to share few dashboards of narrower time frame instead of one dashboard covering the whole time frame desired to be shared.


          Issue Links



              aleksandar.yanakiev Aleksandar Yanakiev
              sergey.kuzmichev Sergey Kuzmichev
              1 Vote for this issue
              7 Start watching this issue



                  Time Tracking

                  Original Estimate - Not Specified
                  Not Specified
                  Remaining Estimate - Not Specified
                  Not Specified
                  Time Spent - 3 hours, 25 minutes
                  3h 25m

                    Smart Checklist