enforce on disk action log for self hosted cluster

[xiang90]

The self hosted etcd cluster is critical to the health of a kubernetes cluster, thus to the operator itself.

The highest priority thing for us is to ensure the reliability of such a cluster. We want to debug EVERY failure cases.

Sometimes, users have troubles with their self hosted clusters, and when it happens they cannot even access their k8s cluster. It is usually hard for them to get the previous logs, especially for their testing/dev clusters without external logging system hooked up. But we need etcd operator log to debug such cases.

To make our debugging life easier, we should write another set of log down to local disk for self hosted cluster for every action it takes.. This is very common tricks for early stage systems that care about debug-ability.

/cc @hongchaodeng

[xiang90]

/cc @coresolve

[xiang90]

how to log:

use a well known host path
append to a log file, never truncate

things to log:

• add new pod (with pod manifest)
• remove pod (with pod name)
• spec change (with old and new spec in human readable format)
• changes in running pod (pod failure)
• any API error or failure

/cc @hongchaodeng

[coresolve]

💯 agree!

[hongchaodeng]

done