Uploaded image for project: 'Percona Monitoring and Management'
  1. Percona Monitoring and Management
  2. PMM-5563

Per-Service and per-Node Annotations

Details

    • New Feature
    • Status: Done
    • Medium
    • Resolution: Done
    • None
    • 2.9.0
    • PMM Admin, PMM ManageD
    • None
    • 1
    • Platform Sprint 18, Platform Sprint 19
    • Yes
    • Yes
    • Yes
    • No

    Description

      The Goal: Let people mark some timestamps on the graphs so they will have some visual marker when they look at the graph.

      Annotation - "sticky note on the Graphs". They are visual only, no analytics or anything else on top of them.

      Use cases and user interactions:

      Case 1: SysAdmin did an update of some system library on the Node and expect this might affect Node or Database performance. He/She want to have a note about this

      As a PMM User, I need to be able to run pmm-admin command and add information about changes I've made as annotation, so that in PMM I'll be able to see this information when I will be looking to OS Dashboards for this Node

      Example:

      pmm-admin annotate [--node] "OS upgrade" [--tags "14.04-20.04"] [--node-name=some-other-node-name]
      

      Note: By default, the annotation should be added to the current Node.

      Case 2: SysAdmin did a restart of Database Server and expect this might affect Database performance. He/She want to have a note about this

      Example:

      pmm-admin annotate [--service] "Service Restart" [--tags "manual,second-tag"] [--service-name=mysql5.6]
      

      Note: we need to pass the Service Name if there more then one Service running on the current node

      Case 3: DevOps need to mark moments when new configuration applied on some Nodes or services, so he/she needs and API to specify Node Name, Service Name Annotation next and additional tags, if needed

      DOD:

      Ability to add annotation related to exact service or node. Annotations added via CLI or API should be visible in grafana.
      Annotations added for certain service should have tag with service name and shouldn't have tag `pmm_annotations`. To the text we should add "(Service Name: Name_of_the _service )"
      Annotations added for certain node should have tag with node name and shouldn't have tag `pmm_annotations`. To the text we should add "(Node Name: Node_name )"

      New flags:

      --service

      --service-name

      --node

      --node-name

      All cases examples:

      annotate "global annotation"
      annotate --service "annotation for current service"
      annotate --service-name=mysql57 "annotation for mysql57"
      annotate --node "annotation for current service"
      annotate --node-name=mysql57 "annotation for mysql57"

       

      If given name doesnt exist then no annotation will be added.

       

      Cases for test:

      --node = current node
      --node-name = node with name
      --node --node-name = node with name
      --node --service-name = current node and service with name
      --node --node-name --service-name = node with name and service with name
      --node --service = current node and all services of current node
      -node --node-name --service - -service-name = service with name and node with name
      --service = all services of current node
      --service-name = service with name
      --service --service-name = service with name
      --service --node-name = all services of current node and node with name
      --service-name --node-name = service with name and node with name
      --service --service-name -node-name = service with name and node with name

       

      Out of scope:

      • Removing and Updating existing annotations. 
      • We don't save any data about annotations in pmm-managed.
      • Grafana UI part
      • Implement any analytics on top of Annotations. For Structured objects, we'll do Events later. Events will be designed to have stronger relations to Inventory model and strict
      • if we add annotations for node we don't have to add annotations for services running on that node

      Suggested implementation:

      1. API
        1. New endpoint in pmm-managed API
          1. Validate if service or node is exists
        2. Create grafana annotation via grafana API with extra tag (service name or node name) (https://grafana.com/docs/grafana/latest/http_api/annotations/)
      2. CLI
        1. Node
          1. If node-name is not set
            1. get current node id from pmm-agent
            2. request node-name from pmm-server
          2. if node-name is set just send request
        2. Service
          1. If service name is not set
            1. get current node id from pmm-agent
            2. request all services for current node from pmm-serve
          2. if node-name is set just send request

       

      Discussion:

      • Ability to add annotation per service
      • is it worth add annotations to All Services on the current node if  Annotation to Node added?  Looks like - yes, because Service "should inherit labels from Node "
        • no
      • Figure out how --service and --current-node will work.

       

      How it can be tested directly through Grafana API:

      • Add an annotation for a node
        # curl 'http://admin:[email protected]/graph/api/annotations' -H 'Content-Type: application/json;charset=UTF-8' --data-binary '{"time":1585053244718,"tags":["pmm-server"],"text":"Outage (Node Name: pmm-server)"}'
        {"id":1,"message":"Annotation added"}
        
      • Check the annotation on dashboards for node pmm-server
      • Add an annotation for a service
        # curl 'http://admin:[email protected]/graph/api/annotations' -H 'Content-Type: application/json;charset=UTF-8' --data-binary '{"time":1585053244718,"tags":["ps_5.7_0.0.0.0_1"],"text":"Alarm!!! (Service Name: ps_5.7_0.0.0_1)"}'
        {"id":2,"message":"Annotation added"}
        
      • Check the annotation on dashboards for service ps_5.7_0.0.0.0_1

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              roma.novikov Roma Novikov
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - Not Specified
                  Not Specified
                  Logged:
                  Time Spent - 1 week, 1 day, 50 minutes
                  1w 1d 50m

                  Smart Checklist