coverage flakiness on source/server

[mathetake]

The coverage on source/server recently has been flaky between 94.4 and 95.1 on unrelated PRs, for example #15210 and #15117.

What I have observed:

• This is somehow deterministic, i.e. on some commits, retesting would not work to go green.
• The root cause sits in source/server/connection_handler_impl.cc
  • by digging into the report, you can see that these lines would not be covered sometimes.

Temporarily the coverage threshold is loosen in #15238 to stop this being a blocker on unrelated PRs. We should not tighten again the corresponding threshold until we resolve this flakiness.

[alyssawilk]

cc @asraa @antoniovicente did we think this had something to do with ENVOY_BUG?

[asraa]

Hmmm that was fixed by https://github.com/envoyproxy/envoy/pull/15104/files, ENVOY_BUGs act as they do in opt mode while running coverage (non-fatal). If I peek at coverage reports, I maybe would have seen a decrement of line number counts after an ENVOY_BUG I know was tested, but I don't see that anymore. (Then again, I don't necessarily trust line numbers on coverage reports)

Was this the same issue you were seeing then?

[alyssawilk]

Ah curses, the ENVOY_BUG issue came up at the last envoy call as a potential cause of the coverage flakiness. If it's not that (given it's fixed) we'll have to do some digging :-(

[alyssawilk]

Also @mathetake thanks so much for not just the quick fix but finding the flaky area.
@ggreenway would you have cycles to take a look and add some deterministic testing for that area?

[lambdai]

I am not fully sure ENVOY_BUG is the root cause. One of the KP of conn handler is that there are too many internal classes. It is hard to write the test case to cover these lines.

This is my attempt toward deterministic #15127

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

[mathetake]

not stale

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted" or "no stalebot". Thank you for your contributions.