Uploaded image for project: 'Percona Monitoring and Management'
  1. Percona Monitoring and Management
  2. PMM-5682

Increase the limit on shared dashboard size

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open
    • Priority: High
    • Resolution: Unresolved
    • Affects Version/s: 2.4.0
    • Fix Version/s: None
    • Component/s: PMM Server
    • Labels:
    • Sprint:
      Platform Sprint 16
    • Needs Review:
      Yes
    • Needs QA:
      Yes
    • Needs Doc:
      No

      Description

      STR:

      1. Add a few MySQL services(4-5) to monitoring
      2. Open MySQL Compare
      3. Select all service names
      4. Expand collapsed chart's sections
      5. Click on Share, select Snapshot tab
      6. Change Timeout to 150s
      7. Click on Share with Percona

      Given result: The snapshot link generation is failed with the next error:

      Original report
      There's currently an implicit limit on the possible size of dashboard being shared with Percona from PMM. The limit is 1MB. Sharing of dashboards is a feature of Grafana, and as explained in https://github.com/grafana/grafana/issues/9835 the limit is tied to client_max_body_size setting in involved nginx servers.

      There are two sides to this problem: "client" (local PMM installation) and "server" (Percona Snapshot-sharing Grafana). client_max_body_size should be increased from default 1MB to some higher value (10MB was discussed) universally.

      1. On PMM installation side, with default client_max_body_size sharing a dashboard with size larger than 1MB results in the following error logged:

      nginx log
      2020/03/30 12:35:47 [error] 47#47: *8350 client intended to send too large body: 1161812 bytes, client: xxx.xxx.xxx.xxx, server: _, request: "POST /graph/api/snapshots HTTP/2.0", host: "***", referrer: "***://xxx/graph/d/mysql-innodb/mysql-innodb-details?orgId=1&from=1584158400000&to=1584208799000&var-interval=$__auto_interval_interval&var-node_name=***&var-crop_host=***&var-service_name=***-mysql&var-version=5.7.28-31-31.41-log&var-crop_service=***-my&var-environment=All&var-cluster=All&var-replication_set=All&var-database=All"
      

      2. With the above worked around by adding client_max_body_size 2m; to the installation's nginx, snapshot sharing is failing on Percona side's nginx, with the following being logged on PMM side grafana.log:

      grafana log
      t=2020-03-30T12:39:01+0000 lvl=eror msg="Failed to create external snaphost" logger=context userId=2 orgId=1 uname=*** error="Create external snapshot response status code 413"
      t=2020-03-30T12:39:01+0000 lvl=eror msg="Request Completed" logger=context userId=2 orgId=1 uname=*** method=POST path=/api/snapshots status=500 remote_addr=127.0.0.1 time_ms=435 size=76 referer="https://***/graph/d/mysql-innodb/mysql-innodb-details?orgId=1&from=1584158400000&to=1584208799000&var-interval=$__auto_interval_interval&var-node_name=***&var-crop_host=***&var-service_name=***-mysql&var-version=5.7.28-31-31.41-log&var-crop_service=***-my&var-environment=All&var-cluster=All&var-replication_set=All&var-database=All"
      

      To reproduce "local" issue (point 1), the easiest way to go is to add client_max_body_size 1k;, so that sharing almost any dashboard will fail. For point 2, sufficiently large dashboard will have to be shared.

      Existing workaround is to share few dashboards of narrower time frame instead of one dashboard covering the whole time frame desired to be shared.

        Smart Checklist

          Attachments

            Issue Links

              Activity

                People

                Assignee:
                Unassigned
                Reporter:
                sergey.kuzmichev Sergey Kuzmichev
                Votes:
                1 Vote for this issue
                Watchers:
                7 Start watching this issue

                  Dates

                  Created:
                  Updated:

                    Time Tracking

                    Estimated:
                    Original Estimate - Not Specified
                    Not Specified
                    Remaining:
                    Remaining Estimate - Not Specified
                    Not Specified
                    Logged:
                    Time Spent - 3 hours, 25 minutes
                    3h 25m