Unable to delete the annotations on pods by deleting them in spec.replsets.annotations field

What happened?

We found it impossible to remove the additional annotations by unsetting the key-value pairs under the field `spec.replsets.annotations`. We first added a label `key: value` under the `spec.replsets.annotations` field, and PSMDB operator correctly adds the annotation to the pods. However, later we wanted to remove the `key: value` label from the pods by deleting it from the CR. However, by doing so, PSMDB operator does not remove the annotations from the pods.

Did you expect to see something different?

We expected that the CR represents the desired state of the application, based on the declarative model of Kubernetes. So we expected that when remove the label from the `spec.replsets.annotations` field, PSMDB operator would automatically remove this annotation from the pods.

How to reproduce it:

1. Deploy PSMDB operator
2. Deploy a minimal MongoDB cluster using minimal.yaml:
    `kubectl apply -f example.yaml`

apiVersion: psmdb.percona.com/v1-12-0
kind: PerconaServerMongoDB
metadata:
name: minimal-cluster
spec:
crVersion: 1.12.0
image: percona/percona-server-mongodb:5.0.7-6
allowUnsafeConfigurations: true
upgradeOptions:
apply: 5.0-never
schedule: "0 2 * * *"
secrets:
users: minimal-cluster
replsets:

• name: rs0
  size: 1
  volumeSpec:
  persistentVolumeClaim:
  resources:
  requests:
  storage: 3Gi
  sharding:
  enabled: true
  configsvrReplSet:
  size: 1
  volumeSpec:
  persistentVolumeClaim:
  resources:
  requests:
  storage: 3Gi
  mongos:
  size: 1

3. Add additional annotations `key1: value1` to replica set by applying the cr1.yaml

apiVersion: psmdb.percona.com/v1-12-0
kind: PerconaServerMongoDB
metadata:
name: minimal-cluster
spec:
crVersion: 1.12.0
image: percona/percona-server-mongodb:5.0.7-6
allowUnsafeConfigurations: true
upgradeOptions:
apply: 5.0-never
schedule: "0 2 * * *"
secrets:
users: minimal-cluster
replsets:

• name: rs0
  annotations:
  key1: value1
  size: 1
  volumeSpec:
  persistentVolumeClaim:
  resources:
  requests:
  storage: 3Gi

sharding:
enabled: true

configsvrReplSet:
size: 1
volumeSpec:
persistentVolumeClaim:
resources:
requests:
storage: 3Gi

mongos:
size: 1

4. Observe that the seed service now has the additional annotation `key1: value1`
5. Delete the additional annotation by applying the example CR again
6. Observe that the seed service still has the label `key1: value1`
    

• Kubernetes cluster kind:  Kind cluster

Anything else we need to know:

The root cause is inside the https://github.com/percona/percona-server-mongodb-operator/blob/88f47b92483e54f4eeb070c46408b5fb42830482/pkg/controller/perconaservermongodb/psmdb_controller.go#L1402 where the annotations in the existing statefulSet are merged into the desired statefulSet. The reason of the merge is because mongodb operator does not want to mess up the annotations that are added by other sources.
A possible fix is to keep track of the annotations that are added by the user using the `spec.replsets.annotations` field, so that when users delete some annotations from the field, mongodb operator can delete these from the statefulSet.