Uploaded image for project: 'Percona Server for MySQL'
  1. Percona Server for MySQL
  2. PS-7114

Enhance crash artifacts (core dumps and stack traces) to provide additional information to the operator

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Done
    • Priority: Medium
    • Resolution: Fixed
    • Affects Version/s: 5.7.x, 8.0.x
    • Fix Version/s: 5.7.31-34, 8.0.21-12
    • Component/s: None
    • Labels:

      Description

      Problem

      When PXC operator crashes, we should be able to have as much information as possible to analyze the crash.

      1. Coredumps - It comes disabled by default at kernel level. In order to enable it we need to ajust kernel variables which require users to run in privileged mode. This won't be an option on multiple vendors.
      2. Unresolved stacktraces - Shipping binary with symbols, this will increase the image size and operators are really sensitive to space constraints

      Proposed Solution

      Unresolved stacktraces

      We decided to print extra information under the stack trace in order to be able to analyze it offline. Since operators may rotate the log, that information should be part of the stack trace. New information is:

      • Print binary buildID.
      • Print MySQL version.

      Coredumps

      •  We are using https://code.google.com/archive/p/google-coredumper/ . The original source code didn't compiled. Zsolt Parragi fixed the issues and created a fork containing the fixes - https://github.com/dutow/coredumper which now has officially moved to https://github.com/Percona-Lab/coredumper
      • The idea is to have it as an experimental feature as a submodule located at Percona-Lab.
      • We will add compiler flags to control when this is enabled or disabled.
      • We will link it statically in order to packaging.
      • The default behavior when code-file is enabled will be relay on kernel core dump.
      • A new variable will be introduce to control if library coredump will be used. This variable will be like log-bin. If specified, it will save core.timestamp file under the datadir. If a path is specified, the coredump will be saved under the specified path.

        Smart Checklist

          Attachments

            Activity

              People

              Assignee:
              marcelo.altmann Marcelo Altmann
              Reporter:
              marcelo.altmann Marcelo Altmann
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - Not Specified
                  Not Specified
                  Logged:
                  Time Spent - 3 weeks, 2 hours, 15 minutes
                  3w 2h 15m