Integrate official PMM 2 docker image with PXCO

Use cases

As a user, I need to have the possibility of using an official and fully supported PMM 2 client docker image without any custom modifications.

As a cloud developer, I need to use the official PMM 2 client docker image to avoid customising it every release and support all these changes. It complicates development and release processes. 

Solution

We need to add the possibility of setting up  PMM 2 client using the environment variables. 
In order to set up pmm-agent PMM_AGENT_SETUP variable should be used. If   PMM_AGENT_SETUP is true pmm-agent setup is called before pmm-agent run.  The pmm-agent can be set up using two approaches: all needed information for the configuration can be provided by flags  or via the environment variables. In our case the second approach should be used.

How to use PMM_AGENT_SETUP variable to set up pmm-agent:
We need to declare all needed environment variables:

Pay attention to the fact that the value of the PMM_AGENT_SETUP_NODE_NAME variable should consist of POD_NAMESPACE+POD_NAME.  The following default values for env variables should be the following:

{
    Name:  "PMM_AGENT_SERVER_ADDRESS",
    Value: pmmServerHost,
},
{
    Name:  "PMM_AGENT_SERVER_USERNAME",
    Value: pmmServerUser,
},
{
    Name: "PMM_AGENT_SERVER_PASSWORD",
    ValueFrom: &corev1.EnvVarSource{
        SecretKeyRef: SecretKeySelector(secrets, "pmmserver"),
    },
}, 
    Name:  "PMM_AGENT_LISTEN_PORT",
    Value: "7777",
},
{
    Name:  "PMM_AGENT_PORTS_MIN",
    Value: "30100",
},
{
    Name:  "PMM_AGENT_PORTS_MAX",
    Value: "30105",
},
{
    Name:  "PMM_AGENT_CONFIG_FILE",
    Value: "/usr/local/percona/pmm2/config/pmm-agent.yaml",
},
{
    Name:  "PMM_AGENT_SETUP_NODE_NAME",
    Value: "$(POD_NAMESPASE)-$(POD_NAME)",
},
{
    Name:  "PMM_AGENT_SETUP",
    Value: "1",
},
{
    Name:  "PMM_AGENT_SETUP_FORCE",
    Value: "1",
},
{
    Name:  "PMM_AGENT_SERVER_INSECURE_TLS",
   Value: "1",
},

As soon as pmm-agent is set up we need to add needed service for PMM 2. In case of PXC it is  mysql and proxysql (if proxysql is used as a proxy for PXC). For this need we should use PMM_AGENT_PRERUN_SCRIPT env variable . If variable is non-empty, entrypoint runs given shell script content with pmm-agent run running in the background.  The value of this var should be the following:

    - name: PMM_AGENT_PRERUN_SCRIPT
      value: “sleep 10; pmm-admin add $(DB_TYPE) --skip-connection-check  --username=$(DB_USER) --password=$(DB_PASSWORD) --cluster=$(DB_CLUSTER)  --service-name=$(PMM_AGENT_SETUP_NODE_NAME) --host=$(DB_HOST) --port=$(DB_PORT) $(DB_ARGS); pmm-admin annotate reboot”

Limitations

We can use only PMM 2 client image , so we will officially support only PMM2 for PXCO.

Work to do

1. Improve PXCO to have possibility of using official PMM 2 docker client.
2. Improve e2e monitoring-2-0 test.
3. Update PMM 2  server k8s helm chart to the latest version.
4. Test the possibility of using a PMM 2 server from another cluster.
5. Check the update of the PMM 2 client.