Infinite Tracing logging connection errors at too high a priority

[newrelic-eheinlein]

Description

A customer has reported seeing a lot of these error messages in the logs when Infinite Tracing is configured:

(8) 2020/08/11 14:23:07.972736 {"level":"error","msg":"trace observer response error","context":{"err":"rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: NO_ERROR"}}

After some investigation by the Infinite Tracing team, it looks like these are harmless, and we should log these as INFO/DEBUG errors (or not log them at all).

Additional context

For New Relic employees - see these two Slack threads:

• https://newrelic.slack.com/archives/C0MQ534AD/p1597249643004300
• https://newrelic.slack.com/archives/CP36ZFEFQ/p1597250722002500

[blakeroberts-wk]

Our go agents with infinite tracing configured have now started getting this log about every 30 seconds per pod/container:

(1) 2020/09/11 19:04:39.113127 {"level":"error","msg":"trace observer response error","context":{"err":"EOF"}}

[Miriam-R]

We're investigating whether this should be remediated on the service side.

[RichVanderwal]

Internal NR thread here.

[RichVanderwal]

After some research, it appears that this error is coming from the HTTP2 layer, and is caused by the Infinite Tracing load balancers. We've decided to set up a reproduction, and to inhibit logging for this specific type of error.

[nr-swilloughby]

We have successfully reproduced this issue and will focus on a code change to squelch it.
(23104) 2021/03/11 16:56:38.094567 {"level":"debug","msg":"trace observer response error","context":{"err":"rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: NO_ERROR"}}

[RichVanderwal]

Hello @blakeroberts-wk , this issue should be fixed the the most recent release of the Go Agent, version 3.11.0. Please try it out and let us know if it works!