Feature request: Add ability to set LOG level for "diagd" aka diagnostic logs

[muhanator]

Please describe your use case / problem.
Our team would like to reduce our logging footprint to reduce our costs as we host our own microservices. Our microservices are now stable and do not require the same verbose output as they used to. Currently, the least "verbose" log output is setting the AES_LOG_LEVEL: to error (shown below).

env:
  AES_LOG_LEVEL: error

Even then, there is still a very high logging footprint from emissary/ambassador itself as
logs of INFO level are being displayed.

2021-08-05 18:29:17 diagd 1.8.1-21-g77f6ec45e-dirty [P146TThreadPoolExecutor-0_29] INFO: 0CE05326-EE6E-4B9D-8613-9A9D2FAD71C1: 10.58.131.14 "GET /metrics" 16ms 200 success
2021-08-05 18:29:21 diagd 1.8.1-21-g77f6ec45e-dirty [P146TThreadPoolExecutor-0_28] INFO: BA21CC1A-BC73-4096-8EA6-2708AA4F6089: 10.58.131.18 "GET /metrics" 13ms 200 success
2021-08-05 18:29:22 diagd 1.8.1-21-g77f6ec45e-dirty [P146TThreadPoolExecutor-0_3] INFO: 05288778-EF15-4A89-B6BC-B5F4AD0DB762: 10.58.131.14 "GET /metrics" 16ms 200 success
2021-08-05 18:29:26 diagd 1.8.1-21-g77f6ec45e-dirty [P146TThreadPoolExecutor-0_22] INFO: 9B2F7A08-FBA8-4CEA-8558-95C938F65B2F: 10.58.131.18 "GET /metrics" 13ms 200 success
2021-08-05 18:29:27 diagd 1.8.1-21-g77f6ec45e-dirty [P146TThreadPoolExecutor-0_24] INFO: F7BA0E95-BCA9-40BD-839B-CDC992CAD796: 10.58.131.14 "GET /metrics" 16ms 200 success
2021-08-05 18:29:31 diagd 1.8.1-21-g77f6ec45e-dirty [P146TThreadPoolExecutor-0_15] INFO: 8CFBC403-7579-4421-86FA-095D4609DC70: 10.58.131.18 "GET /metrics" 14ms 200 success
2021-08-05 18:29:32 diagd 1.8.1-21-g77f6ec45e-dirty [P146TThreadPoolExecutor-0_23] INFO: 53E9D092-58C4-4D0A-9A04-D14B7604C4F4: 10.58.131.14 "GET /metrics" 15ms 200 success
2021-08-05 18:29:36 diagd 1.8.1-21-g77f6ec45e-dirty [P146TThreadPoolExecutor-0_8] INFO: 6F401082-76C2-4A1A-B3FB-7BDE8495D4C0: 10.58.131.18 "GET /metrics" 16ms 200 success
2021-08-05 18:29:37 diagd 1.8.1-21-g77f6ec45e-dirty [P146TThreadPoolExecutor-0_28] INFO: 51C0FE1F-A92F-4C2F-8F8B-2607810D6A35: 10.58.131.14 "GET /metrics" 17ms 200 success
2021-08-05 18:29:40 diagd 1.8.1-21-g77f6ec45e-dirty [P146TThreadPoolExecutor-0_18] INFO: 4DCAE8C6-C0FD-4C40-9CDA-2C325CC01984: 10.58.131.18 "GET /metrics" 14ms 200 success
2021-08-05 18:29:42 diagd 1.8.1-21-g77f6ec45e-dirty [P146TThreadPoolExecutor-0_20] INFO: 4E89C44C-5793-45C4-91B4-EE3205C1902E: 10.58.131.14 "GET /metrics" 17ms 200 success

Describe the solution you'd like
My team would like the situation to be the complete opposite where we no longer see the "diagd" logs being displayed by setting its log level to what we want. By default currently, the "diagd" logs are all set to INFO (https://github.com/emissary-ingress/emissary/blob/master/python/ambassador_diag/diagd.py#L97). We just want the ability to be able to set the log level for "diagd" aka the diagnostics logs (which cannot be set from AES_LOG_LEVEL)

Describe alternatives you've considered
We've spoken extensively to the Datawire OSS emissary team on their slack channel (https://www.getambassador.io/docs/edge-stack/latest/topics/running/ambassador/#envoy) about this issue and they requested me to make this feature request.

Additional context
We believe this could be suitable for other teams as well to bring down their logging costs once their microservices have become stable.

[efunk]

Resolved in Edge Stack 2.0.3 EA release.