argocd logs are all treated as errors when run in GKE

[jsoref]

If you are trying to resolve an environment-specific issue or have a one-off question about the edge case that does not require a feature then please consider asking a question in argocd slack channel.

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

all of argocd's logging which is sent to stderr (and is mostly informational) is treated by google cloud's log viewer as errors

See https://stackoverflow.com/questions/57888588/logging-to-stderr-and-stdout-golang-google-cloud-platform

Possible fixes:

• https://github.com/znly/logrus-gce
• https://github.com/apsystole/log

To Reproduce

1. run argocd in gke
2. Visit https://console.cloud.google.com/logs/query;query=resource.labels.container_name%3D%22application-controller%22
3. Look at the logs

Expected behavior

Most items shouldn't be marked as errors w/ !!

Screenshots

Version

{
    "Version": "v1.9.0+00f53b5",
    "BuildDate": "2021-01-27T00:04:02Z",
    "GitCommit": "00f53b510d87750492d8446e3303f6ab28258bce",
    "GitTreeState": "clean",
    "GoVersion": "go1.14.12",
    "Compiler": "gc",
    "Platform": "linux/amd64",
    "KsonnetVersion": "v0.13.1",
    "KustomizeVersion": "v3.8.1 2020-07-16T00:58:46Z",
    "HelmVersion": "v3.4.1+gc4e7485",
    "JsonnetVersion": "v0.17.0"
}

Logs

Paste any relevant application logs here.

[jannfis]

I believe this not only applicable to GKE, but to many other log processors as well. As you correctly noted, we send everything to stderr, while (by convention), only errors should be sent there.

I guess a proper fix would be to send all messages with a severity lower than error to stdout. I believe we could do it without changing the logger library we use, but that's something to investigate.

[jannfis]

Basically, this is a dupe of #3491