Ambassador pods displaying log level info DESPITE setting AES_LOG_LEVEL to error and envoy_log_path to null

[muhanator]

Describe the bug
I am trying to set the log level of my Ambassador deployment to error. I changed the env variable AES_LOG_LEVEL to error as specified in (https://www.getambassador.io/docs/edge-stack/latest/topics/running/aes-extensions/#aes_log_level). I also set "envoy_log_path: /dev/null" for my module as specified in (https://www.getambassador.io/docs/edge-stack/latest/topics/running/ambassador/#envoy). But neither of these changes change the log output of my pod to level error. The level remains info.

To Reproduce
Steps to reproduce the behavior:

1. Change env variable AES_LOG_LEVEL to error on your Ambassador chart.

env:
  AES_LOG_LEVEL: error

2. set "envoy_log_path: /dev/null" for your ambassador module in your service.

service:
  externalTrafficPolicy: 'Local'
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-internal: 'true'
    getambassador.io/config: |
      ---
      apiVersion: ambassador/v1
      kind: Module
      name: ambassador
      config:
        service_port: 8443
        use_remote_address: true
        buffer:  
          max_request_bytes: 10485760
        x_num_trusted_hops: 1
        x_forwarded_proto_redirect: true
        grpc_stats:
          upstream_stats: true
          all_methods: true
        envoy_log_path: /dev/null
    external-dns.alpha.kubernetes.io/hostname: '${serviceVariable.ambassador_domain}'

3. After adding these changes, deploy it.

Expected behavior
Only Logs of error level should be displayed.

Actual behavior
Logs of info level are being displayed.

2021-08-05 18:29:17 diagd 1.8.1-21-g77f6ec45e-dirty [P146TThreadPoolExecutor-0_29] INFO: 0CE05326-EE6E-4B9D-8613-9A9D2FAD71C1: 10.58.131.14 "GET /metrics" 16ms 200 success
2021-08-05 18:29:21 diagd 1.8.1-21-g77f6ec45e-dirty [P146TThreadPoolExecutor-0_28] INFO: BA21CC1A-BC73-4096-8EA6-2708AA4F6089: 10.58.131.18 "GET /metrics" 13ms 200 success
2021-08-05 18:29:22 diagd 1.8.1-21-g77f6ec45e-dirty [P146TThreadPoolExecutor-0_3] INFO: 05288778-EF15-4A89-B6BC-B5F4AD0DB762: 10.58.131.14 "GET /metrics" 16ms 200 success
2021-08-05 18:29:26 diagd 1.8.1-21-g77f6ec45e-dirty [P146TThreadPoolExecutor-0_22] INFO: 9B2F7A08-FBA8-4CEA-8558-95C938F65B2F: 10.58.131.18 "GET /metrics" 13ms 200 success
2021-08-05 18:29:27 diagd 1.8.1-21-g77f6ec45e-dirty [P146TThreadPoolExecutor-0_24] INFO: F7BA0E95-BCA9-40BD-839B-CDC992CAD796: 10.58.131.14 "GET /metrics" 16ms 200 success
2021-08-05 18:29:31 diagd 1.8.1-21-g77f6ec45e-dirty [P146TThreadPoolExecutor-0_15] INFO: 8CFBC403-7579-4421-86FA-095D4609DC70: 10.58.131.18 "GET /metrics" 14ms 200 success
2021-08-05 18:29:32 diagd 1.8.1-21-g77f6ec45e-dirty [P146TThreadPoolExecutor-0_23] INFO: 53E9D092-58C4-4D0A-9A04-D14B7604C4F4: 10.58.131.14 "GET /metrics" 15ms 200 success
2021-08-05 18:29:36 diagd 1.8.1-21-g77f6ec45e-dirty [P146TThreadPoolExecutor-0_8] INFO: 6F401082-76C2-4A1A-B3FB-7BDE8495D4C0: 10.58.131.18 "GET /metrics" 16ms 200 success
2021-08-05 18:29:37 diagd 1.8.1-21-g77f6ec45e-dirty [P146TThreadPoolExecutor-0_28] INFO: 51C0FE1F-A92F-4C2F-8F8B-2607810D6A35: 10.58.131.14 "GET /metrics" 17ms 200 success
2021-08-05 18:29:40 diagd 1.8.1-21-g77f6ec45e-dirty [P146TThreadPoolExecutor-0_18] INFO: 4DCAE8C6-C0FD-4C40-9CDA-2C325CC01984: 10.58.131.18 "GET /metrics" 14ms 200 success
2021-08-05 18:29:42 diagd 1.8.1-21-g77f6ec45e-dirty [P146TThreadPoolExecutor-0_20] INFO: 4E89C44C-5793-45C4-91B4-EE3205C1902E: 10.58.131.14 "GET /metrics" 17ms 200 success

[efunk]

Resolved in Edge Stack 2.0.3 EA release.

[ronenl3]

@efunk I'm using this image (the one used in your Helm chart) and still see this issue on latest image (1.14.2)

[muhanator]

@ronenl3 , that is because the issue was fixed in v2.0.3-ea --> https://github.com/emissary-ingress/emissary/blob/v2.0.3-ea/python/ambassador_diag/diagd.py#L99-L100

The fix was not added in 1.14.2. Need 2.0.3 to be pushed to docker hub @efunk.

[kflynn]

@muhanator The 2.0.3 image is, indeed, pushed: docker.io/datawire/emissary:2.0.3-ea 🙂

[ronenl3]

I don't understand, I'm using the docker.io/datawire/ambassador image, is it going to get this update?
What is the difference between this image and the emissary image?