The Goal: Let people mark some timestamps on the graphs so they will have some visual marker when they look at the graph.

Annotation - "sticky note on the Graphs". They are visual only, no analytics or anything else on top of them.

Use cases and user interactions:

Case 1: SysAdmin did an update of some system library on the Node and expect this might affect Node or Database performance. He/She want to have a note about this

As a PMM User, I need to be able to run pmm-admin command and add information about changes I've made as annotation, so that in PMM I'll be able to see this information when I will be looking to OS Dashboards for this Node

Example:

pmm-admin annotate [--node] "OS upgrade" [--tags "14.04-20.04"] [--node-name=some-other-node-name]

Note: By default, the annotation should be added to the current Node.

Case 2: SysAdmin did a restart of Database Server and expect this might affect Database performance. He/She want to have a note about this

Example:

pmm-admin annotate [--service] "Service Restart" [--tags "manual,second-tag"] [--service-name=mysql5.6]

Note: we need to pass the Service Name if there more then one Service running on the current node

Case 3: DevOps need to mark moments when new configuration applied on some Nodes or services, so he/she needs and API to specify Node Name, Service Name Annotation next and additional tags, if needed

DOD:

Ability to add annotation related to exact service or node. Annotations added via CLI or API should be visible in grafana.
Annotations added for certain service should have tag with service name and shouldn't have tag `pmm_annotations`. To the text we should add "(Service Name: Name_of_the _service )"
Annotations added for certain node should have tag with node name and shouldn't have tag `pmm_annotations`. To the text we should add "(Node Name: Node_name )"

New flags:

--service

--service-name

--node

--node-name

All cases examples:

annotate "global annotation"
annotate --service "annotation for current service"
annotate --service-name=mysql57 "annotation for mysql57"
annotate --node "annotation for current service"
annotate --node-name=mysql57 "annotation for mysql57"

If given name doesnt exist then no annotation will be added.

Cases for test:

--node = current node
--node-name = node with name
--node --node-name = node with name
--node --service-name = current node and service with name
--node --node-name --service-name = node with name and service with name
--node --service = current node and all services of current node
-node --node-name --service - -service-name = service with name and node with name
--service = all services of current node
--service-name = service with name
--service --service-name = service with name
--service --node-name = all services of current node and node with name
--service-name --node-name = service with name and node with name
--service --service-name -node-name = service with name and node with name

Out of scope:

• Removing and Updating existing annotations. 
• We don't save any data about annotations in pmm-managed.
• Grafana UI part
• Implement any analytics on top of Annotations. For Structured objects, we'll do Events later. Events will be designed to have stronger relations to Inventory model and strict
• if we add annotations for node we don't have to add annotations for services running on that node

Suggested implementation:

1. API
   1. New endpoint in pmm-managed API
      1. Validate if service or node is exists
   2. Create grafana annotation via grafana API with extra tag (service name or node name) (https://grafana.com/docs/grafana/latest/http_api/annotations/)
2. CLI
   1. Node
      1. If node-name is not set
         1. get current node id from pmm-agent
         2. request node-name from pmm-server
      2. if node-name is set just send request
   2. Service
      1. If service name is not set
         1. get current node id from pmm-agent
         2. request all services for current node from pmm-serve
      2. if node-name is set just send request

Discussion:

• Ability to add annotation per service
• is it worth add annotations to All Services on the current node if  Annotation to Node added?  Looks like - yes, because Service "should inherit labels from Node "
  • no
• Figure out how --service and --current-node will work.

How it can be tested directly through Grafana API:

• Add an annotation for a node

  # curl 'http://admin:admin@localhost/graph/api/annotations' -H 'Content-Type: application/json;charset=UTF-8' --data-binary '{"time":1585053244718,"tags":["pmm-server"],"text":"Outage (Node Name: pmm-server)"}'
  {"id":1,"message":"Annotation added"}
  

• Check the annotation on dashboards for node pmm-server

• Add an annotation for a service

  # curl 'http://admin:admin@localhost/graph/api/annotations' -H 'Content-Type: application/json;charset=UTF-8' --data-binary '{"time":1585053244718,"tags":["ps_5.7_0.0.0.0_1"],"text":"Alarm!!! (Service Name: ps_5.7_0.0.0_1)"}'
  {"id":2,"message":"Annotation added"}
  

• Check the annotation on dashboards for service ps_5.7_0.0.0.0_1